Domain-Adaptive Pretraining pada Model Bahasa Besar untuk Peningkatan Pemahaman Pengetahuan Keamanan Siber

Muhammad, Aditya Rizki (2026) Domain-Adaptive Pretraining pada Model Bahasa Besar untuk Peningkatan Pemahaman Pengetahuan Keamanan Siber. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5025221272-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (2MB) | Request a copy

Abstract

Transformasi digital telah mendorong peningkatan pemanfaatan Large Language Models (LLM) dalam berbagai sektor, termasuk keamanan siber yang membutuhkan kemampuan analisis informasi secara cepat dan akurat. Meskipun LLM modern menunjukkan kemampuan generalisasi yang baik, model yang dilatih menggunakan korpus umum masih mengalami domain gap ketika diterapkan pada pengetahuan keamanan siber, khususnya Cyber Threat Intelligence (CTI), sehingga berdampak pada penurunan performa pada pemahaman tugastugas berbasis pengetahuan spesifik domain. Berbagai pendekatan domain adaptation, seperti Supervised Fine-tuning (SFT), Retrieval-Augmented Generation (RAG), dan Continual Pretraining (CPT), telah dikembangkan untuk mengatasi permasalahan tersebut, namun sebagian besar memerlukan sumber daya komputasi yang besar. Penelitian ini bertujuan mengevaluasi efektivitas CPT sebagai metode adaptasi domain keamanan siber yang tetap layak diterapkan pada lingkungan dengan sumber daya komputasi terbatas. Eksperimen dilakukan menggunakan empat model open-source berukuran 3 hingga 8 miliar parameter, dataset pelatihan keamanan siber publik sebanyak 40 juta token, serta infrastruktur satu GPU NVIDIA A100. Evaluasi dilakukan menggunakan benchmark CTI-MCQ, CyberMetric, dan SecEval untuk mengukur kemampuan model dalam memahami pengetahuan Cyber Threat Intelligence maupun keamanan siber secara umum. Hasil penelitian menunjukkan bahwa CPT mampu meningkatkan performa pada sebagian besar model dan benchmark yang digunakan. Peningkatan yang konsisten diperoleh pada benchmark CTI-MCQ dan CyberMetric, sedangkan pada SecEval sebagian besar model juga mengalami peningkatan meskipun terdapat satu konfigurasi model yang mengalami sedikit penurunan pada beberapa subdomain. Selain itu, proses pelatihan dapat diselesaikan hanya menggunakan satu GPU dengan kebutuhan VRAM maksimum sekitar 21 GB, jauh lebih rendah dibandingkan beberapa penelitian sebelumnya yang memerlukan konfigurasi multi-GPU dan jumlah token pelatihan yang jauh lebih besar. Temuan ini menunjukkan bahwa CPT berbasis dataset keamanan siber berukuran relatif kecil tetap mampu meningkatkan pemahaman pengetahuan keamanan siber pada LLM sekaligus menawarkan pendekatan yang lebih efisien dan mudah direproduksi pada lingkungan dengan keterbatasan sumber daya komputasi.
=====================================================================================================================================

Digital transformation has accelerated the adoption of Large Language Models (LLMs) across various domains, including cybersecurity, where timely and accurate analysis of security related information is essential. Although modern LLMs demonstrate strong generalization capabilities, models pre-trained on general-purpose corpora still suffer from a domain gap when applied to cybersecurity knowledge, particularly Cyber Threat Intelligence (CTI), resulting in degraded performance on domain-specific knowledge tasks. Several domain adaptation approaches, including Supervised Fine-Tuning (SFT), Retrieval-Augmented Generation (RAG), and Continual Pre-training (CPT), have been proposed to address this limitation. However, most existing approaches require substantial computational resources. This study evaluates the effectiveness of CPT as a domain adaptation method for cybersecurity while maintaining feasibility in resource-constrained environments. Experiments were conducted using four open-source LLMs ranging from 3 to 8 billion parameters, a public cybersecurity corpus consisting of 40 million training tokens, and a single NVIDIA A100 GPU. Model performance was evaluated using the CTI-MCQ, CyberMetric, and SecEval benchmarks to assess both Cyber Threat Intelligence and general cybersecurity knowledge. The experimental results demonstrate that CPT consistently improves performance across most models and evaluation benchmarks. Consistent performance gains were observed on the CTI-MCQ and CyberMetric benchmarks, while most models also achieved improvements on SecEval, despite a slight performance decline in one model configuration on several subdomains. Furthermore, the entire training process was completed using a single GPU with a maximum VRAM requirement of approximately 21 GB, significantly lower than previous studies that relied on multi-GPU configurations and substantially larger training corpora. These findings indicate that CPT with a relatively small cybersecurity corpus can effectively enhance the cybersecurity knowledge understanding of LLMs while providing a computationally efficient and reproducible approach for cybersecurity domain adaptation in resource-constrained settings.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Large Language Model, Cyber Security, Fine-tuning, Large Language Model, Cyber Security, Fine-tuning
Subjects:	T Technology > T Technology (General) > T58.8 Productivity. Efficiency T Technology > TA Engineering (General). Civil engineering (General) > TA1637 Image processing--Digital techniques. Image analysis--Data processing.
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering > 55201-(S1) Undergraduate Thesis
Depositing User:	Aditya Rizki Muhammad
Date Deposited:	25 Jul 2026 12:12
Last Modified:	25 Jul 2026 12:12
URI:	http://repository.its.ac.id/id/eprint/137425

Actions (login required)

View Item