Implementasi Sistem Asisten Kepatuhan ISO/IEC 27001 Вerbasis Retrieval-Augmented Generation Dengan Studi Kasus Pada ITS Dan UNRAM

Wicaksono, Rahmad Aji (2026) Implementasi Sistem Asisten Kepatuhan ISO/IEC 27001 Вerbasis Retrieval-Augmented Generation Dengan Studi Kasus Pada ITS Dan UNRAM. Other thesis, Institut Teknologi Sepuluh Nopember.

Text (Buku TA 5027221034) 5027221034-Thesis_Undergraduate.pdf - Accepted Version Download (2MB)

Abstract

Penelitian ini berfokus pada dua tantangan utama implementasi Large Language Model (LLM) untuk mendukung kepatuhan standar ISO/IEC 27001 di data center ITS dan UNRAM: risiko halusinasi jawaban yang tidak terikat pada bukti (evidence-bounded) dan ketiadaan instrumen evaluasi standar. Untuk mengatasinya, penelitian ini mengusulkan Sistem Asisten Kepatuhan berbasis Retrieval-Augmented Generation (RAG) yang governance-ready menggunakan pendekatan Design Science Research (DSR). Sistem ini mengintegrasikan tiga pilar utama: (1) Arsitektur Controlled RAG Document Version untuk mengelola hierarki kebijakan dan memastikan jawaban terikat pada bukti yang berlaku; (2) Compliance Benchmark Suite khusus domain ISO 27001; dan (3) Audit-Ready Answer Scoring Rubric (ARASR) untuk mengevaluasi kualitas keluaran dari perspektif tata kelola. Hasil pengujian komparatif terhadap empat model LLM menunjukkan bahwa arsitektur Governance-Ready RAG secara signifikan meningkatkan skor kesiapan audit (audit-readiness) rata-rata sebesar 46,15% dibandingkan baseline RAG. Model Qwen 2.5 (14B) terbukti sebagai mesin inferensi terbaik dengan skor tertinggi 4,1 (dari skala 5) dan persentase kegagalan kritis yang sangat rendah, berkat keberhasilannya mengeksekusi mekanisme penolakan aman (safe refusal). Intervensi filter tata kelola terbukti efektif menyelamatkan model dari disorientasi versi dokumen usang. Selain itu, instrumen ARASR berhasil tervalidasi secara statistik dengan reliabilitas antar-penilai yang sangat kuat (Krippendorff’s a = 0,86). Secara keseluruhan, prototipe sistem ini terbukti tangguh, andal dalam menekan risiko halusinasi regulasi, dan memiliki transferabilitas tinggi antar institusi, sehingga berkontribusi nyata pada penguatan tata kelola AI untuk audit ISMS.
=====================================================================================================================================
This research focuses on two main challenges in the implementation of Large Language Models (LLM) to support ISO/IEC 27001 standard compliance in the data centers of ITS and UNRAM: the risk of hallucinated answers that are not evidence-bounded, and the absence of a standard evaluation instrument. To address these issues, this study proposes a governance-ready Compliance Assistant System based on Retrieval-Augmented Generation (RAG) using the Design Science Research (DSR) approach. The system integrates three main pillars: (1) a Controlled RAG Document Version Architecture to manage policy hierarchies and ensure answers are bound to valid evidence; (2) an ISO 27001 domain-specific Compliance Benchmark Suite; and (3) an Audit-Ready Answer Scoring Rubric (ARASR) to evaluate output quality from a governance perspective. Comparative testing results on four LLM models demonstrate that the Governance-Ready RAG architecture significantly increased the average audit-readiness score by 46.15% compared to the baseline RAG. The Qwen 2.5 (14B) model proved to be the best inference engine with the highest score of 4.1 (out of a 5-point scale) and a very low critical error rate, owing to its success in executing the safe refusal mechanism. The governance filter intervention proved effective in saving the models from disorientation caused by obsolete document versions. Furthermore, the ARASR instrument was statistically validated with a very strong inter-rater reliability (Krippendorff’s a = 0.86). Overall, this system prototype has proven to be robust, reliable in suppressing the risk of regulatory hallucinations, and possesses high transferability across institutions, thereby contributing significantly to the strengthening of AI governance for ISMS audits.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Retrieval-Augmented Generation, ISO/IEC 27001, Tata Kelola AI, Audit-Readiness, Kepatuhan Data Center, AI Governance, Data Centre Compliance.
Subjects:	T Technology > T Technology (General) > T58.5 Information technology. IT--Auditing
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Information Technology > 59201-(S1) Undergraduate Thesis
Depositing User:	Rahmad Aji Wicaksono
Date Deposited:	03 Aug 2026 03:31
Last Modified:	03 Aug 2026 03:31
URI:	http://repository.its.ac.id/id/eprint/141648

Actions (login required)

View Item