Development of an LLM- and RAG-Based Chatbot for Academic and Student Affairs Services in the Department of Information Systems at ITS

Damanik, Morin Adepatrick (2026) Development of an LLM- and RAG-Based Chatbot for Academic and Student Affairs Services in the Department of Information Systems at ITS. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5026201088-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (5MB) | Request a copy

Abstract

Kebutuhan akan akses informasi akademik dan kemahasiswaan yang cepat, akurat, dan mudah ditelusuri terus meningkat, sementara informasi tersebut masih tersebar pada berbagai peraturan, panduan, silabus, dan dokumen layanan digital. Penelitian ini mengembangkan proof of concept chatbot berbasis Large Language Model (LLM) dan Retrieval-Augmented Generation (RAG) untuk membantu mahasiswa dan dosen memperoleh informasi di Departemen Sistem Informasi, Institut Teknologi Sepuluh Nopember. Sistem dikembangkan sebagai aplikasi web berbasis Flask. Lima dokumen PDF inti digunakan sebagai korpus evaluasi, yaitu Peraturan Akademik ITS 2025, Buku Petunjuk Teknis Magang ITS 2020, Peraturan SKEM ITS 2023, Silabus DSI Kurikulum 2023, dan FAQ Layanan Digital Utama ITS. Beberapa berkas JSON digunakan sebagai data terstruktur pendukung pada implementasi, tetapi tidak dimasukkan ke dalam evaluasi RAGAS agar pengujian berfokus pada pipeline retrieval dokumen PDF. Dataset evaluasi terdiri atas 30 pasangan pertanyaan dan jawaban acuan yang ditinjau oleh dua pakar, sehingga menghasilkan 60 keputusan validasi. Evaluasi teknis menggunakan RAGAS menghasilkan skor faithfulness 0,977738, answer relevancy 0,662012, context precision 0,892747, dan context recall 0,966667. Hasil tersebut menunjukkan kinerja awal yang kuat pada dataset terbatas dan mendukung kelayakan sistem sebagai proof of concept. Namun, pengujian dengan pertanyaan pengguna yang lebih banyak dan beragam tetap diperlukan sebelum penerapan operasional.
========================================================================================================================================
The need for fast, accurate, and traceable access to academic and student affairs information motivated the development of an intelligent information assistant for the Department of Information Systems at Institut Teknologi Sepuluh Nopember (ITS). This study developed a web-based chatbot that combines a Large Language Model (LLM) with Retrieval-Augmented Generation (RAG). The application was implemented using Flask and retrieves evidence from five core PDF documents covering academic regulations, internship guidance, SKEM regulations, the 2023 Information Systems curriculum syllabus, and ITS digital-service FAQs. Supporting JSON files provide selected structured information, such as course lecturer mappings, but were excluded from the main RAGAS evaluation. The evaluation dataset contained 30 question–answer pairs. Two domain experts reviewed the same 30 items, producing 60 expert judgments before the final dataset was reconciled. Technical evaluation using RAGAS produced scores of 0,977738 for faithfulness, 0,662012 for answer relevancy, 0,892747 for context precision, and 0,966667 for context recall. These results indicate strong initial performance on the bounded, expert-validated test set and support the feasibility of the chatbot as an early-stage proof of concept. A larger and more diverse set of independent real-user queries is still required before operational deployment.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	chatbot, Large Language Model, Retrieval-Augmented Generation, RAGAS, layanan akademik ================ chatbot, Large Language Model, Retrieval-Augmented Generation, RAGAS, academic services
Subjects:	Q Science > QA Mathematics > QA336 Artificial Intelligence Q Science > QA Mathematics > QA76.758 Software engineering Z Bibliography. Library Science. Information Resources > ZA Information resources > Z699.5 Information storage and retrieval systems
Divisions:	Faculty of Information and Communication Technology > Information Systems > 57201-(S1) Undergraduate Thesis
Depositing User:	Morin Adepatrick Damanik
Date Deposited:	03 Aug 2026 04:31
Last Modified:	03 Aug 2026 04:31
URI:	http://repository.its.ac.id/id/eprint/142045

Actions (login required)

View Item