Fuadi, Mukhlish (2026) Kombinasi Percakapan Closed Domain dan Open Domain pada Conversational AI. Doctoral thesis, Institut Teknologi Sepuluh Nopember.
![[thumbnail of 7022231007-Doctoral.pdf]](http://repository.its.ac.id/style/images/fileicons/text.png) |
Text
7022231007-Doctoral.pdf Restricted to Repository staff only Download (6MB) | Request a copy |
Abstract
Pemanfaatan Large Language Model (LLM) dalam layanan konsumen sering kali terkendala oleh risiko halusinasi pada domain khusus, kebutuhan infrastruktur komputasi yang besar, serta isu privasi data. Di sisi lain, sistem berbasis closed domain konvensional belum mampu menangani interaksi informal yang muncul secara alami dalam komunikasi manusia. Penelitian ini mengusulkan arsitektur Conversational AI hybrid berbasis pipeline adaptif untuk bahasa Indonesia yang mengintegrasikan mekanisme intent routing, semantic retrieval, dan verifikasi faktual guna mengombinasikan keunggulan sistem closed domain dan open domain berbasis compact LLM 3B. Sistem yang diusulkan terdiri atas empat komponen modular utama, yaitu SLM Router (Router-ID) dengan mekanisme dual-threshold, Semantic Similarity Check (SSC) dan retrieval Hybrid+Reranker berbasis Embed-ID, serta GLiNER-ID untuk perhitungan Entity Support Score (ESS) sebagai mekanisme verifikasi faktual pasca generasi. Seluruh komponen dikembangkan menggunakan compact encoder yang dioptimalkan melalui language-aware vocabulary pruning, menghasilkan reduksi ukuran model sebesar 60% tanpa degradasi performa yang signifikan. Evaluasi pada domain layanan akademik menunjukkan bahwa sistem yang diusulkan mengungguli baseline Naive RAG dengan Composite Score sebesar 0,764 dibandingkan 0,646. Pada tahap routing, sistem mencapai Intent Routing Accuracy sebesar 91,8% dibandingkan 88,2% pada baseline serta meningkatkan akurasi penanganan kueri out-of-scope dari 56,1% menjadi 80,1% , menunjukkan kemampuan yang lebih baik dalam mengenali batas domain dan memilih jalur respons yang sesuai. Dengan total latensi 31,8% lebih rendah dibandingkan baseline, hasil penelitian ini menunjukkan bahwa kombinasi routing adaptif, retrieval semantik, dan verifikasi faktual berbasis entitas mampu mengintegrasikan kemampuan closed domain dan open domain secara lebih akurat dan lebih cepat.
=========================================================================================================================================
The use of Large Language Models (LLMs) in customer service is often hindered by the risk of hallucinations in specialized domains, the need for significant computational infrastructure, and data privacy concerns. On the other hand, conventional closed-domain systems are not yet capable of handling the informal interactions that naturally arise in human communication. This study proposes a hybrid Conversational AI architecture for the Indonesian language, built on an adaptive pipeline that integrates intent routing, semantic retrieval, and factual verification mechanisms, combining the strengths of closed-domain and open-domain systems with a compact 3B LLM. The proposed system consists of four main modular components: the SLM Router (Router-ID) with a dual-threshold mechanism, the Embed-ID-based Semantic Similarity Check (SSC) and Hybrid+Reranker retrieval system, and GLiNER-ID for computing the Entity Support Score (ESS) as a post-generation factual verification mechanism. All components were developed using a compact encoder optimized via language-aware vocabulary pruning, resulting in a 60% reduction in model size with minimal performance degradation. Evaluation in the academic services domain shows that the proposed system outperforms the Naive RAG baseline with a Composite Score of 0.764 compared to 0.646. In the routing stage, the system achieved an Intent Routing Accuracy of 91.8% compared to 88.2% for the baseline and improved out-of-scope query-handling accuracy from 56.1% to 80.1%, demonstrating a stronger ability to recognize domain boundaries and select appropriate response paths. With total latency 31.8% lower than the baseline, these results demonstrate that the combination of adaptive routing, semantic retrieval, and entity-based factual verification can integrate closed-domain and open-domain capabilities more accurately and more quickly.
| Item Type: | Thesis (Doctoral) |
|---|---|
| Uncontrolled Keywords: | chatbot, conversational ai, intent routing, named entity recognition, retrieval augmented generation |
| Subjects: | Q Science > Q Science (General) > Q325.5 Machine learning. Support vector machines. Q Science > QA Mathematics > QA336 Artificial Intelligence Q Science > QA Mathematics > QA76.87 Neural networks (Computer Science) |
| Divisions: | Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Electrical Engineering > 20001-(S3) PhD Thesis |
| Depositing User: | Mukhlish Fuadi |
| Date Deposited: | 30 Jul 2026 02:15 |
| Last Modified: | 30 Jul 2026 02:15 |
| URI: | http://repository.its.ac.id/id/eprint/140077 |
Actions (login required)
 |
View Item |