Sistem Interaksi Berbasis Input Multimodal Pada Platform Virtual Try-On Kiosk

Febrina, Vania Bunga (2026) Sistem Interaksi Berbasis Input Multimodal Pada Platform Virtual Try-On Kiosk. Diploma thesis, Institut Teknologi Sepuluh Nopember.

Text 5024221069-Undergraduate_Thesis_1.pdf - Accepted Version Restricted to Repository staff only Download (21MB) | Request a copy

Abstract

Perkembangan interaksi manusia-komputer (HCI) mendorong kebutuhan sistem yang lebih alami dan intuitif, terutama pada platform Virtual Try-On (VTO) kiosk yang bertujuan meningkatkan pengalaman pelanggan di industri ritel. Keterbatasan interaksi unimodal konvensional, seperti hanya sentuhan atau gestur, seringkali mengurangi fleksibilitas dan kepuasan pengguna. Penelitian ini bertujuan untuk merancang dan mengembangkan sebuah sistem interaksi berbasis input multimodal yang mengintegrasikan suara, pose tangan, dan sentuhan pada platform VTO kiosk. Metodologi penelitian mengusulkan alur kerja yang terdiri dari empat tahap utama: akuisisi input interaksi, pemrosesan mode interaksi, manajemen user flow, dan visualisasi. Pada tahap pemrosesan, perintah suara dalam Bahasa Indonesia diinterpretasikan menggunakan pendekatan hibrida yang menggabungkan model bahasa IndoBERT untuk perintah kompleks dan sistem berbasis aturan rule-based untuk perintah sederhana. Fokus utama penelitian terletak pada implementasi user flow yang dinamis dengan pendekatan isolasi mode (mode isolation), di mana pengguna secara eksplisit memilih satu mode interaksi aktif pada halaman utama. Pendekatan ini memastikan bahwa sumber daya komputasi dialokasikan secara optimal pada satu pipeline pemrosesan, sehingga menghasilkan responsivitas dan stabilitas sistem yang tinggi. Hasil pengujian terhadap 376 unit percobaan menunjukkan bahwa mode pose mencapai akurasi 87,28% pada jarak ideal 75-100 cm, mode sentuh memberikan latensi terendah (2,29 ms), dan model IndoBERT mencapai akurasi klasifikasi intensi di atas 95% untuk perintah terdaftar. Evaluasi System Usability Scale (SUS) menghasilkan skor rata-rata 76,67 (kategori Good/Acceptable) untuk sistem multimodal terintegrasi. Penelitian ini memberikan kontribusi pada penerapan model bahasa IndoBERT dalam domain HCI dan menjadi referensi untuk pengembangan kios interaktif di masa depan.
=================================================================================================================================
The development of human-computer interaction (HCI) drives the need for more natural and intuitive systems, especially in Virtual Try-On (VTO) kiosk platforms that aim to improve customer experience in the retail industry. The limitations of conventional unimodal interactions, such as only touch or gestures, often reduce flexibility and user satisfaction. This research designed and developed a multimodal input-based interaction system that integrates voice, hand pose, and touch on a VTO kiosk platform. The research methodology follows a workflow consisting of four main stages: interaction input acquisition, interaction mode processing, user flow management, and visualization. In the processing stage, Indonesian voice commands are interpreted using a hybrid approach that combines the IndoBERT language model for complex commands and a rule-based system for simple commands. The system implements a mode isolation approach, where the user explicitly selects one active interaction mode on the main page, ensuring that computing resources are allocated optimally to a single processing pipeline. Testing across 376 trial units demonstrated that pose mode achieved 87.28% accuracy at the ideal distance of 75-100 cm, touch mode provided the lowest latency (2.29 ms), and the IndoBERT model achieved over 95% intent classification accuracy for registered commands. System Usability Scale (SUS) evaluation yielded an average score of 76.67 (Good/Acceptable category) for the integrated multimodal system. This research contributes to the application of the IndoBERT language model in the HCI domain and serves as a reference for the development of interactive kiosks in the future.

Item Type:	Thesis (Diploma)
Uncontrolled Keywords:	Sistem Multimodal, Virtual Try-On (VTO), Interaksi Manusia Komputer (HCI), IndoBERT, Kontrol Suara, Interaksi Pose Tangan, UI/UX Multimodal, Multimodal Systems, Virtual Try-On (VTO), Human-Computer Interaction (HCI), IndoBERT, Voice Control, Hand Pose Interaction, Multimodal UI/UX.
Subjects:	Q Science > QA Mathematics > QA76.9 Computer algorithms. Virtual Reality. Computer simulation. Q Science > QA Mathematics > QA76.9.U83 Graphical user interfaces. User interfaces (Computer systems)--Design. T Technology > T Technology (General) > T59.7 Human-machine systems. T Technology > TA Engineering (General). Civil engineering (General) > TA1637 Image processing--Digital techniques. Image analysis--Data processing.
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Computer Engineering > 90243-(S1) Undergraduate Thesis
Depositing User:	Vania Bunga Febrina
Date Deposited:	24 Jul 2026 02:45
Last Modified:	24 Jul 2026 02:45
URI:	http://repository.its.ac.id/id/eprint/136777

Actions (login required)

View Item