Question Answering System Layanan Informasi Digital Perpustakaan Kota Surabaya Dengan Metode Retrieval-Augmented Generation (RAG)

Putri, Brigita Naraduhita Primanti (2025) Question Answering System Layanan Informasi Digital Perpustakaan Kota Surabaya Dengan Metode Retrieval-Augmented Generation (RAG). Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5027211055_Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (3MB) | Request a copy

Abstract

Layanan informasi Perpustakaan Kota Surabaya saat ini masih tersebar di berbagai platform dan bersifat konvensional sehingga pengunjung yang bertanya harus menunggu respons admin sesuai jam operasional. Sistem rekomendasi dan pencarian buku juga masih terbatas, hanya berdasarkan judul secara exact yang menyebabkan pengalaman pengguna kurang optimal. Oleh karena itu, untuk mengatasi permasalahan tersebut diusulkan pengembangan aplikasi Question Answering System (QAS) "SobatJoyo!" yang dirancang untuk menyediakan beberapa informasi, meliputi informasi umum dan sejarah Perpustakaan Kota Surabaya, tanya jadwal libur, dan rekomendasi buku. Aplikasi ini menggunakan metode Retrieval-Augmented Generation (RAG) dengan penyimpanan data multicollection untuk menjawab beberapa jenis pertanyaan dan metode retriever Hybrid Rerank antara Cross-Encoder dan Bi-Encoder untuk meningkatkan akurasi jawaban. Dataset dikumpulkan dari hasil wawancara dengan staf Perpustakaan Balai Pemuda dan Perpustakaan Rungkut, data dari Dinas Perpustakaan dan Kearsipan Kota Surabaya (Dispusip. Penelitian ini dilakukan dengan menguji beberapa pendekatan retriever, yaitu Bi-Encoder, Cross-Encoder, kombinasi Bi-Encoder dan Cross-Encoder yang mengambil indeks pertama, serta metode Hybrid Rerank. Evaluasi sistem dilakukan menggunakan metode RAGAS (Retrieval-Augmented Generation Assessment Score) serta user testing untuk mengukur tingkat keakuratan dan kelayakan aplikasi. Hasil penelitian menunjukkan bahwa metode retrieval Hybrid Rerank dengan komposisi rasio 40% Bi-encoder dan 60% Cross-encoder menghasilkan performa yang lebih unggul dibandingkan metode Cross-encoder murni maupun metode kombinasi indeks pertama dari cross-encoder dan Bi-encoder. Metode Hybrid Rerank menghasilkan skor evaluasi metrik Context Precision (0,877), Context Recall (0,833), Faithfulness (0,879). Selain itu, dari total 100 pertanyaan, hanya 7 pertanyaan yang tidak terjawab oleh sistem yang menunjukkan tingkat keberhasilan yang cukup tinggi. Metode ini juga menunjukkan efisiensi dengan berhasil mereduksi jumlah token hingga 45% dibandingkan dengan metode Bi-encoder murni. Diharapkan hasil ini dapat meningkatkan kemudahan akses informasi terkait layanan Perpustakaan Kota Surabaya.
====================================================================================================================================
The information services of the Surabaya City Library are currently scattered across various platforms and remain conventional, requiring visitors to wait for responses from the admin during operational hours. The existing book recommendation and search systems are also limited, relying solely on exact title matches, which results in a suboptimal user experience. To address these issues, this study proposes the development of a Question Answering System (QAS) called "SobatJoyo” is designed to provide several information, including general library information, historical background, holiday schedules, and book recommendations. The application adopts the Retrieval-Augmented Generation (RAG) framework with multi-collection data storage to handle diverse question types, utilizing a Hybrid Reranking retriever that integrates Cross-Encoder and Bi-Encoder approaches to enhance answer accuracy. The dataset was compiled from interviews with staff from the Balai Pemuda and Rungkut Library branches, as well as data from the Surabaya City Library and Archives Service (Dispusip). Several retriever approaches were evaluated, including Bi-Encoder, Cross-Encoder, a combined index selection method, and the proposed Hybrid Rerank method. System evaluation was conducted using RAGAS (Retrieval-Augmented Generation Assessment) and user testing to assess accuracy and usability. The results show that the Hybrid Rerank method, with a 40% Bi-Encoder and 60% Cross-Encoder ratio, outperformed the pure Cross-Encoder and index selection methods. This method achieved evaluation scores of Context Precision (0.877), Context Recall (0.833), and Faithfulness (0.879). Moreover, only 7 out of 100 questions were unanswered, indicating a high success rate. The Hybrid Rerank method also demonstrated efficiency by reducing token usage by up to 45% compared to the pure Bi-Encoder approach. These findings are expected to improve access to information related to Surabaya City Library services.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Bi-Encoder, Cross-Encoder, Hybrid Rerank, Question Answering System (QAS), Retrieval-Augmented Generation Assessment Score (RAGAS), Retrieval-Augmented Generation (RAG).
Subjects:	Q Science > QA Mathematics > QA336 Artificial Intelligence
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Information Technology > 59201-(S1) Undergraduate Thesis
Depositing User:	Brigita Naraduhita Primanti Putri
Date Deposited:	23 Jul 2025 03:59
Last Modified:	23 Jul 2025 03:59
URI:	http://repository.its.ac.id/id/eprint/120742

Actions (login required)

View Item