Klasifikasi Emosi pada Lagu Berbahasa Indonesia dengan Input Multimodal Berbasis Transformer

Najiihah, Farra Sabiilah An (2026) Klasifikasi Emosi pada Lagu Berbahasa Indonesia dengan Input Multimodal Berbasis Transformer. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5025221284-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (9MB) | Request a copy

Abstract

Lagu memiliki kemampuan untuk menyampaikan emosi dan membangkitkan perasaan tertentu pada pendengarnya. Berkembangnya platform musik digital mendorong kebutuhan akan sistem pengenalan emosi musik (Music Emotion Recognition/MER) yang akurat dan efisien. Penelitian ini mengembangkan model klasifikasi emosi lagu berbahasa Indonesia menggunakan pendekatan multimodal berbasis Transformer dengan perbandingan menggunakan teknik Early Fusion dan Late Fusion yang mengintegrasikan modalitas audio dan lirik. Penelitan ini menggunakan dataset sebanyak 1.500 lagu pop berbahasa Indonesia yang terbagi ke dalam empat kelas emosi yaitu tenang, sedih, senang dan marah, berdasarkan model emosi Russel. Fitur audio direpresentasikan menggunakan MFCC, Chroma, Spectral Centroid, dan RMS sementara fitur lirik diekstraksi menggunakan model pretrained IndoBERT. Kedua representasi digabungkan melalui teknik fusion (Early Fusion dan Late Fusion) sebelum diproses lebih lanjut untuk menghasilkan prediksi emosi. Model dievaluasi menggunakan skema Stratified 5-Fold Cross Validation dengan Matthews Correlation Coefficient (MCC) sebagai metrik evaluasi utama. Hasil penelitian menunjukkan bahwa modalitas lirik memberikan kontribusi paling dominan, dengan MCC rata-rata sebesar 0,48, dibandingkan modalitas audio yang hanya mencapai MCC rata-rata terbaik sebesar 0,11. Pada skenario multimodal, Late Fusion menghasilkan performa terbaik dengan MCC rata-rata sebesar 0,47, mengungguli Early Fusion yang memperoleh MCC rata-rata sebesar 0,41. Evaluasi tambahan melalui human voting terhadap 96 lagu menunjukkan bahwa persepsi emosi berdasarkan audio tidak selalu konsisten dengan label berbasis lirik, dengan tingkat kesepakatan antar-responden yang tergolong lemah hingga sedang (Fleiss' Kappa = 0,231 fair agreement). Penelitian ini menyimpulkan bahwa pendekatan Late Fusion lebih efektif dibandingkan Early Fusion dalam mengintegrasikan informasi multimodal untuk pengenalan emosi musik pada lagu pop berbahasa Indonesia.
=================================================================================================================================
Songs have the ability to convey emotions and evoke certain feelings in their listeners. The growth of digital music platforms has driven the need for an accurate and efficient Music Emotion Recognition (MER) system. This research develops an emotion classification model for Indonesian-language songs using a Transformer-based multimodal approach, comparing Early Fusion and Late Fusion techniques that integrate audio and lyric modalities. This research uses a dataset of 1.500 Indonesian pop songs divided into four emotion classes, namely calm, sad, happy, and angry, based on Russell's model of affect. Audio features are represented using MFCC, Chroma, Spectral Centroid, and RMS, while lyric features are extracted using the pretrained IndoBERT model. Both representations are combined through a fusion technique (Early Fusion and Late Fusion) before being further processed to produce emotion predictions. The model is evaluated using a Stratified 5-Fold Cross Validation scheme with the Matthews Correlation Coefficient (MCC) as the primary evaluation metric. The results show that the lyric modality provides the most dominant contribution, with an average MCC of 0,48, compared to the audio modality, which only achieves a best average MCC of 0,11. In the multimodal scenario, Late Fusion produces the best performance with an average MCC of 0,47, outperforming Early Fusion, which obtains an average MCC of 0,41. An additional evaluation through human voting on 96 songs showed that emotion perception based on audio was not always consistent with lyric-based labels, with inter-rater agreement classified as weak to moderate (Fleiss' Kappa = 0.231 fair agreement). This research concludes that the Late Fusion approach is more effective than Early Fusion in integrating multimodal information for music emotion recognition in Indonesian-language pop songs.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Musik Emotion Recognition, Transformer, Early Fusion, Late Fusion, multimodal, lagu berbahasa Indonesia,Music Emotion Recognition, Transformer, Early Fusion, Late Fusion, multimodal, Indonesian language songs
Subjects:	M Music and Books on Music > ML Literature of music
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering > 55201-(S1) Undergraduate Thesis
Depositing User:	Farra Sabiilah Annajiihah
Date Deposited:	27 Jul 2026 03:11
Last Modified:	27 Jul 2026 03:11
URI:	http://repository.its.ac.id/id/eprint/137716

Actions (login required)

View Item