Automated Essay Scoring Berbahasa Indonesia Menggunakan Hybrid Retrieval-Augmented Generation dan Contrastive In-Context Learning

Suwida, Katon (2026) Automated Essay Scoring Berbahasa Indonesia Menggunakan Hybrid Retrieval-Augmented Generation dan Contrastive In-Context Learning. Masters thesis, Institut Teknologi Sepuluh Nopember.

Text 6025221039-Master_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (3MB) | Request a copy

Abstract

Perkembangan Natural Language Processing (NLP) mendorong pemanfaatan Automated Essay Scoring (AES) untuk meningkatkan efisiensi penilaian. Namun, penerapan langsung Large Language Model (LLM) pada AES masih menghadapi tantangan berupa halusinasi, miskalibrasi, dan kesulitan mematuhi standar penilaian pada domain spesifik. Penelitian ini mengusulkan pipeline AES berbahasa Indonesia yang mengintegrasikan Hybrid Retrieval-Augmented Generation (Hybrid RAG), Contrastive In-Context Learning (CICL), dan Parameter-Efficient Fine-Tuning melalui Low-Rank Adaptation (LoRA). Sistem mengombinasikan BM25, IndoBERT-FAISS, Keyword Boosting, Cross-Encoder Reranker, dan Focus Mode untuk mengoptimalkan relevansi konteks. Eksperimen dilakukan menggunakan model Qwen3-4B dengan membandingkan pendekatan zero-shot dan fine-tuning LoRA. Evaluasi kinerja menggunakan ablation study pada 16 konfigurasi dengan dataset berisi 312 esai Biologi siswa MTsN 2 Kota Kediri. Hasil penelitian menunjukkan bahwa penggabungan langsung berbagai metode retrieval menyebabkan context dilution, yang menurunkan kinerja model zero-shot. Pada kondisi tersebut, penerapan CICL berfungsi sebagai acuan rubrik yang meningkatkan kinerja model zero-shot hingga mencapai Quadratic Weighted Kappa (QWK) 0,7914. Sebaliknya, model LoRA mencapai kinerja tertinggi dengan QWK 0,8554 dan menunjukkan toleransi yang lebih baik terhadap variasi konteks serta bahasa siswa. Temuan ini mengindikasikan bahwa efektivitas CICL bergantung pada pendekatan inferensi; CICL memberikan peningkatan yang signifikan pada model zero-shot, namun kontribusinya pada model LoRA terbatas karena pengetahuan domain telah tersimpan dalam bobot model selama proses fine-tuning. Secara keseluruhan, pipeline yang diusulkan terbukti efektif dalam mengintegrasikan mekanisme retrieval dan kalibrasi rubrik untuk penilaian esai Biologi berbahasa Indonesia.
==================================================================================================================================
Automated Essay Scoring (AES) improves the efficiency of essay assessment, but the direct application of Large Language Models (LLMs) faces challenges such as hallucinations, score miscalibration, and difficulty adhering to domain-specific grading criteria. This study proposes an Indonesian AES pipeline that integrates Hybrid Retrieval-Augmented Generation (Hybrid RAG), Contrastive In-Context Learning (CICL), and Parameter-Efficient Fine-Tuning via Low-Rank Adaptation (LoRA). The system combines BM25, IndoBERT-FAISS, Keyword Boosting, a Cross-Encoder Reranker, and Focus Mode to optimize contextual relevance. Experiments utilized the Qwen3-4B model, comparing zero-shot inference and LoRA-based fine-tuning. Evaluation involved an ablation study with 16 configurations on a dataset of 312 Indonesian biology essays from MTsN 2 Kota Kediri. Results indicate that directly combining multiple retrieval methods in Hybrid RAG causes context dilution, which decreases the zero-shot model's performance. In this scenario, CICL functions as a rubric reference, increasing the zero-shot model's Quadratic Weighted Kappa (QWK) to 0.7914. Conversely, the LoRA model achieved the highest performance (QWK 0.8554) and demonstrated greater tolerance to contextual variations and linguistic diversity in student responses. Findings indicate that CICL effectiveness depends on the inference approach. While CICL significantly improves zero-shot performance, its contribution to the LoRA model is marginal because domain knowledge is already embedded in the model weights during fine-tuning. Overall, the proposed pipeline effectively integrates retrieval mechanisms and rubric calibration for assessing Indonesian biology essays.

Item Type:	Thesis (Masters)
Uncontrolled Keywords:	Automated Essay Scoring, Contrastive In-Context Learning, Hybrid Retrieval, Large Language Model, LoRA, Retrieval-Augmented Generation
Subjects:	T Technology > T Technology (General) T Technology > T Technology (General) > T57.5 Data Processing T Technology > T Technology (General) > T58.62 Decision support systems
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering > 55101-(S2) Master Thesis
Depositing User:	Katon Suwida
Date Deposited:	04 Aug 2026 07:03
Last Modified:	04 Aug 2026 07:03
URI:	http://repository.its.ac.id/id/eprint/143301

Actions (login required)

View Item