Rekomendasi Sitasi Paper dengan Validasi Dua Tahap Menggunakan SciBERT dan Ensemble LLM

Pratama, Muhammad Fikri Sandi (2026) Rekomendasi Sitasi Paper dengan Validasi Dua Tahap Menggunakan SciBERT dan Ensemble LLM. Masters thesis, Institut Teknologi Sepuluh Nopember.

[thumbnail of 6025241021-Master_Thesis.pdf] Text
6025241021-Master_Thesis.pdf - Accepted Version
Restricted to Repository staff only

Download (11MB) | Request a copy

Abstract

Jumlah publikasi ilmiah yang terus bertambah setiap tahun membuat proses pencarian referensi yang relevan menjadi semakin sulit. Sistem rekomendasi sitasi yang ada umumnya bekerja dalam satu tahap dan bergantung pada satu model, serta belum memiliki mekanisme validasi yang keandalannya dapat diukur. Penelitian ini mengusulkan sistem validasi sitasi dua tahap. Tahap pertama menggunakan SciBERT untuk menyaring pasangan chunk antar paper berdasarkan skor cosine similarity dengan threshold 0,95. Tahap kedua memvalidasi kandidat hasil penyaringan menggunakan ensemble lima Large Language Model (LLM), yaitu LLaMA3, Qwen2.5, Gemma2, Mixtral, dan Dolphin-Mixtral, dengan mekanisme majority voting minimal tiga dari lima model. Dataset yang digunakan terdiri dari 128 publikasi IEEE tahun 2019-2024 yang dibagi menjadi chunk berukuran 150 kata dengan overlap 30 kata. Karena tidak menggunakan ground truth, evaluasi dilakukan menggunakan Cohen's Kappa, Fleiss' Kappa, stabilitas leave-one-out, dan distribusi vote, serta perbandingan efisiensi dan kualitas validasi terhadap pendekatan single-stage. Hasil eksperimen menunjukkan bahwa tahap pertama mereduksi 6.890.990 pasangan chunk menjadi 252 kandidat, dan tahap kedua menghasilkan 158 pasangan VALID (62,70%). Tingkat kesepakatan antar model tergolong fair dengan rata-rata Cohen's Kappa 0,3115 dan Fleiss' Kappa 0,2321. Walaupun demikian, keputusan ensemble tetap stabil dengan rata-rata stabilitas leave-one-out 92,86%, dan 81,01% pasangan VALID didukung oleh minimal empat model. Pendekatan dua tahap hanya membutuhkan 1.260 inferensi LLM dengan waktu 3 jam 24 menit, sedangkan validasi tanpa penyaringan membutuhkan 34.454.950 inferensi dengan estimasi waktu sekitar 10 tahun. Skor similarity SciBERT hanya memiliki korelasi lemah dengan jumlah vote (ρ = 0,290), sehingga tahap validasi ensemble diperlukan untuk memilah kandidat yang memiliki skor similarity hampir sama. Dengan demikian, sistem validasi dua tahap membuat validasi sitasi menggunakan ensemble LLM dapat dijalankan dalam waktu yang wajar, serta menghasilkan keputusan yang tingkat keyakinan dan reliabilitasnya dapat diukur.
========================================================================================================================================
The growing number of scientific publications every year makes it increasingly difficult to find relevant references. Existing citation recommendation systems generally work in a single stage and rely on a single model, without a validation mechanism whose reliability can be measured. This research proposes a two-stage citation validation system. The first stage uses SciBERT to filter chunk pairs across papers based on cosine similarity with a threshold of 0.95. The second stage validates the filtered candidates using an ensemble of five Large Language Models (LLMs), namely LLaMA3, Qwen2.5, Gemma2, Mixtral, and DolphinMixtral, with a majority voting rule of at least three out of five models. The dataset consists of 128 IEEE publications from 2019-2024, split into 150-word chunks with a 30-word overlap. Since no ground truth is used, the evaluation employs Cohen's Kappa, Fleiss' Kappa, leave-one-out stability, and vote distribution, together with a comparison of efficiency and validation quality against the single-stage approach. The results show that the first stage reduces 6,890,990 chunk pairs to 252 candidates, and the second stage produces 158 VALID pairs (62.70%). At the paper level, these VALID pairs yield 37 citation recommendations from 26 source papers to 25 recommended papers. The agreement among models is fair, with an average Cohen's Kappa of 0.3115 and a Fleiss' Kappa of 0.2321. Nevertheless, the ensemble decisions remain stable with an average leave-one-out stability of 92.86%, and 81.01% of the VALID pairs are supported by at least four models. The two-stage approach requires only 1,260 LLM inferences completed in 3 hours 24 minutes, whereas validation without filtering would require 34,454,950 inferences with an estimated time of about 10 years. The SciBERT similarity score has only a weak correlation with the number of votes (ρ = 0.290), so the ensemble validation stage is needed to separate candidates with nearly identical similarity scores. Therefore, the two-stage validation system makes LLM ensemble-based citation validation feasible within a reasonable time and produces decisions whose confidence level and reliability can be measured.

Item Type: Thesis (Masters)
Uncontrolled Keywords: ensemble LLM, inter-rater agreement, majority voting, SciBERT, validasi sitasi
Subjects: Q Science > Q Science (General) > Q325.5 Machine learning. Support vector machines.
Q Science > QA Mathematics > QA336 Artificial Intelligence
Q Science > QA Mathematics > QA76.9.I58 Recommender systems (Information filtering)
Divisions: Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering > 55101-(S2) Master Thesis
Depositing User: Muhammad Fikri Sandi Pratama
Date Deposited: 16 Nov 2026 02:32
Last Modified: 16 Nov 2026 02:32
URI: http://repository.its.ac.id/id/eprint/146061

Actions (login required)

View Item View Item