Perancangan Sistem Backend Berbasis LLM dan RAG untuk Chatbot dan Laporan Monitoring dan Evaluasi ITS

Dewati, Jeany Aurellia Putri (2026) Perancangan Sistem Backend Berbasis LLM dan RAG untuk Chatbot dan Laporan Monitoring dan Evaluasi ITS. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5027221008-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (9MB) | Request a copy

Abstract

Sistem monitoring dan evaluasi (Monev) di Institut Teknologi Sepuluh Nopember (ITS) berperan penting dalam memantau capaian kinerja institusi, namun pengelolaan data yang tersebar dan kompleks sering menyulitkan proses penyusunan laporan serta pencarian informasi oleh pengguna. Penelitian ini merancang sistem backend berbasis Large Language Model (LLM) dan Retrieval-Augmented Generation (RAG) untuk mendukung penyusunan laporan Monev dan menyediakan layanan chatbot asisten berbasis data internal ITS. Sistem dikembangkan dengan memanfaatkan workflow orchestrator n8n untuk mengintegrasikan basis data institusional dengan vector database Qdrant, serta menjalankan model secara lokal untuk menjaga keamanan dan privasi data. Penelitian ini membandingkan dua model embedding , yaitu BGE-M3 dan Snowflake, serta dua model bahasa, yaitu Qwen 2.5 dan SeaLLM. Hasil pengujian menunjukkan bahwa penggunaan embedding BGE-M3 memberikan performa yang lebih baik dibandingkan Snowflake dalam proses retrieval data berbahasa Indonesia. Berdasarkan pengujian User Acceptance Testing (UAT), kombinasi BGE-M3 dan Qwen 2.5 memperoleh skor kepuasan tertinggi sebesar 3,0 dari skala 5 dan tingkat akurasi jawaban chatbot sebesar 65%. Meskipun akurasi yang diperoleh belum sepenuhnya optimal, hasil tersebut menunjukkan bahwa sistem mampu memberikan jawaban yang relatif relevan dan membantu pengguna dalam mengakses informasi Monev. Temuan ini mengindikasikan bahwa pemilihan model embedding dan LLM yang tepat berpengaruh signifikan terhadap kualitas sistem chatbot berbasis RAG pada konteks data institusional.
================================================================================================================================
Monitoring and evaluation (Monev) systems at Institut Teknologi Sepuluh Nopember (ITS) play an important role in assessing institutional performance. However, the complexity and distribution of internal data often hinder efficient report generation and information retrieval. This study proposes a backend system based on Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) to support Monev report generation and provide a chatbot assistant using internal ITS data. The system is implemented using n8n as a workflow orchestrator to integrate institutional databases with the Qdrant vector database, while all models are deployed locally to ensure data security and privacy. This research evaluates two embedding models, BGE-M3 and Snowflake, and two language models, Qwen 2.5 and SeaLLM. Experimental results indicate that BGE-M3 outperforms Snowflake in retrieving Indonesian-language institutional data. Based on User Acceptance Testing (UAT), the combination of BGE-M3 and Qwen 2.5 achieves the highest user satisfaction score of 3.0 on a five-point Likert scale and a chatbot answer accuracy of 65%. Although the accuracy has not reached an optimal level, the system demonstrates its capability to provide relevant responses and assist users in accessing Monev-related information. These findings highlight the importance of appropriate model selection in improving the performance of RAG-based chatbot systems for institutional data.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Large Language Model, RAG, n8n, Chatbot, Monev ITS, BGE-M3, Qwen 2.5,Large Language Model, RAG, n8n, Chatbot, ITS Monev, BGE-M3, Qwen 2.5
Subjects:	T Technology > T Technology (General) > T58.62 Decision support systems
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Information Technology > 59201-(S1) Undergraduate Thesis
Depositing User:	Jeany Aurellia Putri Dewati
Date Deposited:	29 Jan 2026 03:28
Last Modified:	29 Jan 2026 03:28
URI:	http://repository.its.ac.id/id/eprint/130918

Actions (login required)

View Item