Optimalisasi Respons Chatbot Konsultasi Hukum Perdata melalui Prompt Engineering pada Arsitektur Retrieval-Augmented Generation (RAG)

Samad, Abdul (2026) Optimalisasi Respons Chatbot Konsultasi Hukum Perdata melalui Prompt Engineering pada Arsitektur Retrieval-Augmented Generation (RAG). Masters thesis, Institut Teknologi Sepuluh Nopember.

Text 6022241093-Master_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (9MB) | Request a copy

Abstract

Informasi hukum perdata seringkali sulit diakses oleh masyarakat awam karena aturan hukumnya yang kompleks dan biaya konsultasi yang cukup mahal. Untuk mengatasi kendala ini, dikembangkan sebuah chatbot konsultasi hukum perdata berbasis arsitektur Retrieval Augmented Generation (RAG) yang memanfaatkan Google Gemini API dan ChromaDB sebagai vector database. Guna meningkatkan kualitas jawaban, diterapkan teknik prompt engineering dengan membandingkan enam strategi: Zero-Shot, Few-Shot, Role-Based, Chain-of-Thought (CoT), Structured Output, dan RAG-Optimized Prompting. Evaluasi dilakukan secara kuantitatif melalui skor faktual, retrieval precision, waktu respons, dan parameter RAGAS, serta secara kualitatif melalui validasi tiga pakar hukum perdata dan pengukuran pengalaman pengguna. Hasil kuantitatif menunjukkan skor faktual rata-rata 84% dengan tingkat kebenaran jawaban 93,3%, waktu respons rata-rata 3,2 detik, dan retrieval precision 83%. Evaluasi otomatis RAGAS menghasilkan skor faithfulness 0,874, answer relevancy 0,844, context precision 0,830, dan context recall 0,796. Di antara keenam strategi, RAG-Optimized Prompting menunjukkan kinerja terbaik dengan skor faktual 84%, meningkat 12 poin dari baseline Zero-Shot (72% menjadi 84%), sedangkan skor relevansinya naik dari 3,2/5,0 menjadi 4,3/5,0. Evaluasi kualitatif oleh pakar memberikan skor rata-rata 3,81/4,00 (kategori Sangat Baik). Responden survei pengguna (n=25) memberikan skor System Usability Scale (SUS) 76,9/100 (kategori Good) dengan tingkat kepuasan rata-rata 4,1/5,0. Hasil ini menegaskan pentingnya pemilihan teknik prompt engineering yang tepat; Ketika diintegrasikan dengan arsitektur RAG, teknik ini mampu menghasilkan jawaban hukum perdata yang akurat dan bermanfaat.

=======================================================================================================================================

Civil law information is often difficult for the general public to access due to its complex legal rules and generally high consultation fees. To address these challenges, a civil law consultation chatbot based on the Retrieval-Augmented Generation (RAG) architecture was developed, utilizing the Google Gemini API and ChromaDB as a vector database. To enhance answer quality, prompt engineering techniques were applied by comparing six strategies: Zero-Shot, Few-Shot, Role-Based, Chain-of-Thought (CoT), Structured Output, and RAGOptimized Prompting. Evaluation was conducted quantitatively through factual scores, retrieval precision, response time, and RAGAS parameters, and qualitatively through validation by three civil law experts and user experience measurement. Quantitative results showed an average factual score of 84% with an answer truthfulness rate of 93.3%, an average response time of 3.2 seconds, and a retrieval precision of 83%. Automated RAGAS evaluation yielded scores of 0.874 for faithfulness, 0.844 for answer relevancy, 0.830 for context precision, and 0.796 for context recall. Among the six strategies, RAG-Optimized Prompting demonstrated the best performance with a factual score of 84%, an improvement of 12 points over the Zero-Shot baseline (72% to 84%), while its relevance score increased from 3.2/5.0 to 4.3/5.0. Qualitative evaluation by experts yielded an average score of 3.81/4.00 (Very Good category). User survey respondents (n=25) provided a System Usability Scale (SUS) score of 76.9/100 (Good category) with an average satisfaction level of 4.1/5.0. These results underscore the importance of selecting appropriate prompt engineering techniques; when integrated with the RAG architecture, they are capable of generating accurate and beneficial civil law answers.

Item Type:	Thesis (Masters)
Uncontrolled Keywords:	Akurasi Faktual, Chatbot Hukum Perdata, Prompt Engineering, RAGAS, Retrieval-Augmented Generation (RAG). Factual Accuracy, Civil Law Chatbot, Prompt Engineering, RAGAS, Retrieval-Augmented Generation (RAG)
Subjects:	Q Science > QA Mathematics > QA336 Artificial Intelligence
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Electrical Engineering > 20101-(S2) Master Thesis
Depositing User:	Abdul Samad
Date Deposited:	02 Aug 2026 03:53
Last Modified:	02 Aug 2026 03:53
URI:	http://repository.its.ac.id/id/eprint/142038

Actions (login required)

View Item