Implementasi Sistem Agentic Retrieval-Augmented Generation pada Guideline dan Knowledge Base Modul SEVIMA Platform

Firas, Fairuuz Azmi (2026) Implementasi Sistem Agentic Retrieval-Augmented Generation pada Guideline dan Knowledge Base Modul SEVIMA Platform. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5025221057-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (3MB) | Request a copy

Abstract

Knowledge base SEVIMA Platform memuat ratusan artikel panduan. Namun, mekanisme pencarian konvensional berbasis kata kunci belum mampu memahami maksud pengguna ketika pertanyaan menggunakan sinonim, bersifat ambigu, atau membutuhkan informasi yang tersebar pada beberapa artikel sekaligus.
Penelitian ini mengimplementasikan sistem Agentic Retrieval-Augmented Generation pada knowledge base dan guideline SEVIMA Platform. Sistem dikembangkan menggunakan arsitektur hybrid yang menggabungkan jalur langsung dan jalur agentic berbasis ReAct yang diorkestrasi menggunakan LangGraph. Arsitektur tersebut dilengkapi router heuristik dan empat tool agen. Dokumen direpresentasikan sebagai vektor berdimensi 3.072 menggunakan model embedding text-embedding-3-large dan disimpan pada vector database Pinecone. Tiga konfigurasi penalaran, yaitu single-step, multi-step, dan adaptive routing, dievaluasi menggunakan tiga model open-weight, yaitu Qwen3 8B, Qwen3 14B, dan Gemma 4 E4B. Evaluasi dilakukan menggunakan 214 pertanyaan riil yang telah tervalidasi. Kualitas keluaran dinilai menggunakan lima metrik RAGAS dengan GPT-4o sebagai LLM-as-a-Judge. Pengujian juga menelaah sensitivitas sistem terhadap parameter retrieval TOP_K serta ketahanan sistem terhadap variasi gaya bahasa pertanyaan.
Hasil pengujian menunjukkan bahwa konfigurasi single-step dengan Qwen3 8B memperoleh kebenaran jawaban tertinggi sebesar 0,714, diikuti konfigurasi adaptive sebesar 0,710. Sementara itu, konfigurasi multi-step memperoleh nilai terendah sebesar 0,665 dengan biaya komputasi paling tinggi. Analisis menunjukkan bahwa kualitas jawaban sangat ditentukan oleh keberhasilan retrieval. Hal ini ditunjukkan oleh korelasi sebesar 0,82 antara context recall dan kebenaran jawaban, serta 47 dari 49 kegagalan yang bersumber dari tahap retrieval. Nilai TOP_K sebesar 5 memberikan kualitas terbaik, dan sistem terbukti tahan terhadap variasi gaya bahasa. Temuan utama penelitian ini menunjukkan bahwa faktor pembatas kualitas jawaban berada pada tahap retrieval, bukan pada kedalaman penalaran. Dengan demikian, penambahan penalaran agentic belum memberikan peningkatan kualitas jawaban yang signifikan pada karakteristik dataset yang digunakan.
=====================================================================================================================================
The public knowledge base of the SEVIMA Platform contains hundreds of help articles. However, its conventional keyword-based search mechanism is unable to accurately capture user intent when questions contain synonyms, are ambiguous, or require information distributed across multiple articles.
This study implements an Agentic Retrieval-Augmented Generation system using the guidelines and knowledge base of the SEVIMA Platform. The system is developed using a hybrid architecture that combines a direct execution path with a ReAct-based agentic path orchestrated by LangGraph. The architecture incorporates a heuristic router and four agent tools. Documents are represented as 3.072-dimensional vectors using the text-embedding-3-large embedding model and stored in the Pinecone vector database. Three reasoning configurations, namely single-step, multi-step, and adaptive routing, were evaluated using three open-weight models: Qwen3 8B, Qwen3 14B, and Gemma 4 E4B. The evaluation was conducted using 214 validated real-world questions. Output quality was assessed using five RAGAS metrics with GPT-4o serving as the LLM-as-a-Judge. The evaluation also examined the sensitivity of the system to the TOP_K retrieval parameter and its robustness to variations in question language style.
The results show that the single-step configuration with Qwen3 8B achieved the highest answer correctness score of 0.714, followed by the adaptive configuration with a score of 0.710. Meanwhile, the multi-step configuration obtained the lowest score of 0.665 with the highest computational cost. The analysis shows that answer quality is largely determined by retrieval success. This is indicated by a correlation of 0.82 between context recall and answer correctness, as well as 47 out of 49 failures originating from the retrieval stage. A TOP_K value of 5 produced the best quality, and the system proved to be robust against variations in language style. The main finding of this study is that the primary limiting factor of answer quality lies in the retrieval stage rather than in reasoning depth. Therefore, incorporating agentic reasoning did not provide a significant improvement in answer quality for the characteristics of the dataset used in this study.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Agentic RAG, knowledge base, Large Language Model, Pinecone, SEVIMA Platform, Agentic RAG, knowledge base, Large Language Model, Pinecone, SEVIMA Platform.
Subjects:	T Technology > T Technology (General) T Technology > T Technology (General) > T58.64 Information resources management Z Bibliography. Library Science. Information Resources > ZA Information resources > Z699.5 Information storage and retrieval systems
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering > 55201-(S1) Undergraduate Thesis
Depositing User:	Fairuuz Azmi Firas
Date Deposited:	21 Jul 2026 07:57
Last Modified:	21 Jul 2026 07:57
URI:	http://repository.its.ac.id/id/eprint/136024

Actions (login required)

View Item