Deteksi Konten Negatif Pada Media Sosial Code-Mixed Indonesia menggunakan Klasifikasi Multi-Label Berbasis Nilai-Nilai Pancasila dan Analisis Sentimen Dengan Fine-Tuned Transformer

Hallatu, Stefani Tasya (2026) Deteksi Konten Negatif Pada Media Sosial Code-Mixed Indonesia menggunakan Klasifikasi Multi-Label Berbasis Nilai-Nilai Pancasila dan Analisis Sentimen Dengan Fine-Tuned Transformer. Masters thesis, Institut Teknologi Sepuluh Nopember.

Text 6025241011-Master_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (6MB) | Request a copy

Abstract

Penelitian ini mengusulkan pendekatan deteksi konten negatif berbasis klasifikasi multi-label nilai-nilai Pancasila yang diintegrasikan dengan analisis sentimen menggunakan model transformer. Dataset dikumpulkan dari berbagai platform media sosial melalui tahapan text preprocessing, zero-shot annotation menggunakan Llama 3:8B via Ollama, expert validation, serta confidence score filtering dengan ambang batas 0,75, sehingga diperoleh 41.138 data berlabel sebagai dataset pelatihan. Validasi dua pakar menghasilkan rata-rata Cohen's Kappa 0,8193 (Almost Perfect Agreement), sedangkan evaluasi terhadap pseudo-label menghasilkan 0,8092, sehingga zero-shot annotation layak digunakan pada kondisi keterbatasan data berlabel. Penelitian ini mengevaluasi empat model transformer (IndoBERT, IndoBERTweet, mBERT, XLM-RoBERTa) dengan tiga strategi fine-tuning (FFT, LoRA, QLoRA), menghasilkan 12 konfigurasi eksperimen. Pada test set, IndoBERTweet dengan FFT dan IndoBERT dengan LoRA meraih Average F1-Score terbaik sebesar 0,77; pada evaluasi Ground Truth, IndoBERTweet dengan QLoRA terbaik dengan Aspect F1-Score 0,74 dan Sentiment F1-Score 0,69. Dari sisi efisiensi, LoRA/QLoRA hanya memperbarui ±2,68 juta parameter (<4% dari total) dan menurunkan ukuran checkpoint dari 421,85-1.060,76 MB menjadi 10,27 MB (turun 97–99%) dibandingkan FFT. Pada inferensi, QLoRA menurunkan kebutuhan memori GPU sekitar 75-78% (dari 1.976-4.570 MB menjadi 434-1.115 MB), dengan selisih Macro F1-Score terhadap test set tidak lebih dari 0,03 poin untuk nilai Pancasila dan 0,08-0,10 poin untuk sentimen. Hasil ini menunjukkan Parameter-Efficient Fine-Tuning (PEFT), khususnya QLoRA, menurunkan kebutuhan parameter, checkpoint, dan memori GPU secara signifikan dengan penurunan performa yang terukur dan relatif kecil dibandingkan FFT, sehingga berpotensi diterapkan sebagai dasar sistem moderasi konten digital media sosial code-mixed Indonesia dengan kebutuhan komputasi lebih rendah.
===================================================================================================================================
This study proposes a Pancasila values-based multi-label negative content detection approach integrated with sentiment analysis using transformer models. The dataset was collected from various social media platforms through text preprocessing, zero-shot annotation using Llama 3:8B via Ollama, expert validation, and confidence score filtering with a threshold of 0.75, resulting in 41,138 labeled instances used as the training dataset. Validation by two domain experts yielded an average Cohen's Kappa of 0.8193 (Almost Perfect Agreement), while evaluation against the pseudo-labels yielded 0.8092, indicating that zero-shot annotation is a viable approach under limited labelled data conditions. This study evaluated four transformer models (IndoBERT, IndoBERTweet, mBERT, XLM-RoBERTa) using three fine-tuning strategies (FFT, LoRA, QLoRA), resulting in 12 experimental configurations. On the test set, IndoBERTweet with FFT and IndoBERT with LoRA achieved the best Average F1-Score of 0.77; on the Ground Truth evaluation, IndoBERTweet with QLoRA performed best with an Aspect F1-Score of 0.74 and a Sentiment F1-Score of 0.69. In terms of efficiency, LoRA/QLoRA updated only ±2.68 million parameters (<4% of the total) and reduced checkpoint size from 421.85-1,060.76 MB to 10.27 MB (a 97-99% reduction) compared with FFT. At inference, QLoRA reduced GPU memory requirements by approximately 75-78% (from 1,976-4,570 MB to 434-1,115 MB), with a Macro F1-Score gap relative to the test set of no more than 0.03 points for Pancasila values and 0.08-0.10 points for sentiment. These results show that Parameter-Efficient Fine-Tuning (PEFT), particularly QLoRA, substantially reduces parameter, checkpoint, and GPU memory requirements with a measurable and relatively small drop in performance compared with FFT, making it a promising basis for a digital content moderation system for Indonesian code-mixed social media with lower computational requirements.

Item Type:	Thesis (Masters)
Uncontrolled Keywords:	analisis sentimen, code-mixed, deteksi konten negatif, fine-tuning, klasifikasi multi-label, media sosial Indonesia, nilai-nilai Pancasila, transformer, sentiment analysis, code-mixed, negative content detection, fine-tuning, multi-label classification, Indonesian social media, Pancasila values, transformer
Subjects:	Q Science > Q Science (General) > Q325.5 Machine learning. Support vector machines. Q Science > QA Mathematics > QA336 Artificial Intelligence Q Science > QA Mathematics > QA75 Electronic computers. Computer science. EDP Q Science > QA Mathematics > QA76.87 Neural networks (Computer Science) Q Science > QA Mathematics > QA76.9.D343 Data mining. Querying (Computer science)
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering > 55101-(S2) Master Thesis
Depositing User:	Stefani Tasya Hallatu
Date Deposited:	27 Jul 2026 01:31
Last Modified:	27 Jul 2026 01:31
URI:	http://repository.its.ac.id/id/eprint/137754

Actions (login required)

View Item