Penilaian Esai Otomatis Berbasis Tata Bahasa dengan Pengodean Fitur Kesalahan dan Fusi Semantik

Andreas, Kevin (2026) Penilaian Esai Otomatis Berbasis Tata Bahasa dengan Pengodean Fitur Kesalahan dan Fusi Semantik. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5025231168-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (4MB) | Request a copy

Abstract

Esai merupakan salah satu bentuk karya tulis yang banyak digunakan untuk mengukur kemampuan menulis. Karena penilaian manual membutuhkan waktu dan rentan terhadap inkonsistensi, penelitian ini mengembangkan Automatic Essay Scoring (AES) berbasis model RoBERTa dengan mengevaluasi kontribusi fitur kesalahan tata bahasa. Tiga fitur, yaitu jumlah kesalahan, kepadatan kesalahan, dan jenis kesalahan, diekstraksi melalui pipeline koreksi otomatis menggunakan FLAN-T5 dan anotasi ERRANT. Fitur tersebut diproyeksikan ke 64 dimensi dan digabungkan dengan representasi RoBERTa. Empat skenario RoBERTa dan lima algoritma machine learning berbasis fitur diuji pada Task 1, Task 2, dan gabungan dari dua dataset IELTS Writing sejumlah 10.020 dan 1.434 esai. Evaluasi menggunakan QWK dan RMSE dengan validasi silang 5-fold, optimasi Optuna, serta uji statistik bootstrap berpasangan dan uji t berpasangan. Hasil menunjukkan bahwa pengaruh fitur tata bahasa relatif kecil dan tidak konsisten. Dari 18 perbandingan terhadap baseline, hanya dua yang meningkatkan QWK secara signifikan, keduanya pada Task 2 Dataset 1, dan sebagian kenaikan tersebut terjadi pada esai yang memiliki kembaran identik. Tiga perbandingan justru menunjukkan penurunan dan 13 tidak menunjukkan perbedaan signifikan. Meskipun skenario seluruh fitur menghasilkan QWK tertinggi pada Task 2 Dataset 2 (0,7581) dan kelompok gabungan Dataset 2 (0,7369), peningkatannya terhadap baseline tidak signifikan. Kepadatan kesalahan juga berkorelasi lebih kuat dengan prediksi baseline RoBERTa (ρ = −0,39) daripada dengan skor acuan (ρ = −0,33) pada gabungan Dataset 1, yang mengindikasikan bahwa informasi fitur sebagian besar telah terwakili oleh model. Model berbasis fitur saja tertinggal 0,13–0,33 QWK dari baseline, meskipun arsitektur seluruh fitur mengungguli reproduksi model penelitian terdahulu (0,7369 vs. 0,7130).
======================================================================================================================================
Essays are one of the most widely used forms of written work for assessing writing ability. Since manual assessment is time-consuming and prone to inconsistency, this study develops an Automatic Essay Scoring (AES) system based on the RoBERTa model by evaluating the contribution of explicit grammatical error features. Three features, namely error count, error density, and error types, are extracted through an automatic correction pipeline using FLAN-T5 followed by ERRANT annotation. These features are projected into a 64-dimensional representation and combined with the RoBERTa representation. Four RoBERTa-based scenarios and five feature-based machine learning algorithms are evaluated on Task 1, Task 2, and combined data from two IELTS Writing datasets containing 10,020 and 1,434 essays, respectively. Evaluation is conducted using QWK and RMSE with 5-fold cross-validation, Optuna hyperparameter optimization, and statistical tests, namely the paired bootstrap and the paired t-test. The results show that the contribution of grammatical error features is relatively small and inconsistent. Among 18 comparisons against the baseline, only two showed a statistically significant improvement in QWK, both on Task 2 of Dataset 1 and partly driven by essays with identical duplicates, while three comparisons showed significant decreases and 13 showed no significant difference. Although the model using all grammatical features achieved the highest QWK on Task 2 of Dataset 2 (0.7581) and the combined group of Dataset 2 (0.7369), the improvements over the baseline were not statistically significant. Error density was also more strongly correlated with the baseline RoBERTa predictions (ρ = −0.39) than with the reference scores (ρ = −0.33) on the combined group of Dataset 1, indicating that much of the information provided by the explicit features was already captured by the model. Feature-only machine learning models lagged behind the baseline by 0.13–0.33 QWK, although the model incorporating all features outperformed the reproduced classification model from previous research (0.7369 vs. 0.7130).

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Penilaian esai otomatis, Fitur tata bahasa, Automated essay scoring, Grammatical features, ERRANT, RoBERTa, Quadratic Weighted Kappa
Subjects:	T Technology > T Technology (General) > T58.6 Management information systems
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering > 55201-(S1) Undergraduate Thesis
Depositing User:	Kevin Andreas
Date Deposited:	06 Nov 2026 04:39
Last Modified:	06 Nov 2026 04:39
URI:	http://repository.its.ac.id/id/eprint/146045

Actions (login required)

View Item