Pengembangan Intent Classification Berbasis IndoBERT pada Transformer-Based Chatbot untuk Rekomendasi Studi dan Jalur Karier Mahasiswa Sistem Informasi ITS

Sirait, Edward Yosafat (2026) Pengembangan Intent Classification Berbasis IndoBERT pada Transformer-Based Chatbot untuk Rekomendasi Studi dan Jalur Karier Mahasiswa Sistem Informasi ITS. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5026221091-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (4MB) | Request a copy

Abstract

Penelitian ini mengembangkan chatbot akademik berbasis Task-Oriented Dialogue System untuk mendukung rekomendasi studi dan jalur karier mahasiswa Program Studi Sistem Informasi ITS. Sistem dibangun menggunakan kerangka kerja RASA dengan mengintegrasikan IndoBERT sebagai komponen intent classification, DIETClassifier untuk ekstraksi entitas, custom action untuk menjalankan rule-based logic, knowledge base SQLite sebagai sumber data faktual, serta Large Language Model (LLM) sebagai lapisan parafrase pada saat runtime. Model final IndoBERT_DatasetC_LR2e5 memperoleh nilai akurasi sebesar 0,8505, Macro F1 sebesar 0,8798, dan Weighted F1 sebesar 0,8502. Pengujian black-box end-to-end dilakukan terhadap 40 skenario UAT yang dinilai secara independen oleh dua anotator. Setelah proses adjudikasi, sebanyak 30 skenario memperoleh status Lulus sehingga tingkat kelulusan akhir sistem mencapai 75,0%. Evaluasi runtime LLM paraphraser menunjukkan bahwa respons hasil parafrase memperoleh rata-rata skor di atas 4 pada metrik Semantic Preservation, Faithfulness, Fluency, dan Readability. Sementara itu, human evaluation berbasis persona memperoleh nilai keseluruhan sebesar 4,33 dari 5. Hasil penelitian menunjukkan bahwa integrasi IndoBERT, RASA, rule-based logic, dan knowledge base mampu menghasilkan chatbot akademik yang dapat memahami maksud pertanyaan mahasiswa serta memberikan respons berbasis data. LLM membantu menghasilkan respons yang lebih natural dan mudah dipahami, tetapi tetap memerlukan mekanisme pengendalian agar tidak mengubah validitas informasi akademik.
=====================================================================================================================================
This study develops an academic chatbot based on a Task-Oriented Dialogue System to support study and career path recommendations for students in the ITS Information Systems Undergraduate Program. The system was developed using the RASA framework by integrating IndoBERT as the intent classification component, DIETClassifier for entity extraction, custom actions to execute rule-based logic, an SQLite knowledge base as the factual data source, and a Large Language Model (LLM) as a runtime paraphrasing layer. The final model, IndoBERT_DatasetC_LR2e5, achieved an accuracy of 0.8505, a Macro F1 score of 0.8798, and a Weighted F1 score of 0.8502. End-to-end black-box testing was conducted on 40 UAT scenarios that were independently assessed by two annotators. Following an adjudication process, 30 scenarios received a Pass status, resulting in a final system pass rate of 75.0%. The runtime LLM paraphraser evaluation showed that the paraphrased responses achieved average scores above 4 for Semantic Preservation, Faithfulness, Fluency, and Readability. Meanwhile, the persona-based human evaluation obtained an overall score of 4.33 out of 5. The results demonstrate that the integration of IndoBERT, RASA, rule-based logic, and a knowledge base enables the chatbot to understand student intents and provide data-driven academic responses. The LLM produces more natural and readable responses, but safeguards remain necessary to prevent changes to the validity of academic information.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Chatbot Akademik, IndoBERT, Intent Classification, LLM Paraphrasing, RASA, Academic Chatbot, IndoBERT, Intent Classification, LLM Paraphrasing, RASA
Subjects:	Q Science > QA Mathematics > QA336 Artificial Intelligence T Technology > T Technology (General) T Technology > T Technology (General) > T59.7 Human-machine systems.
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Information System > 57201-(S1) Undergraduate Thesis
Depositing User:	Edward Yosafat Sirait
Date Deposited:	30 Jul 2026 06:38
Last Modified:	30 Jul 2026 06:38
URI:	http://repository.its.ac.id/id/eprint/138914

Actions (login required)

View Item