Rancang Bangun Sistem Pemandu Tunanetra Dalam Mengoperasikan Dispenser Air Dengan Large Language Model

Silalahi, Aloysius Abraham (2022) Rancang Bangun Sistem Pemandu Tunanetra Dalam Mengoperasikan Dispenser Air Dengan Large Language Model. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5022221203-Undergraduate_Thesis.pdf Restricted to Repository staff only Download (2MB) | Request a copy

Abstract

Pengguna tunanetra sering kali menghadapi kendala orientasi spasial saat berinteraksi dengan peralatan rumah tangga modern. Salah satu kendala utama ditemukan pada pengoperasian dispenser air galon bawah yang mayoritas menggunakan antarmuka datar tanpa penanda taktil, sehingga memunculkan risiko keselamatan seperti salah menekan tombol air panas. Penelitian ini mengusulkan perancangan Sistem Vision sebagai alat bantu navigasi spasial nirsentuh yang mengintegrasikan pemrosesan bahasa alami, penglihatan komputer 3 dimensi, dan aktuasi wearable. Sistem ini memanfaatkan antarmuka Speech-to-Text dan Large Language Model (LLM) untuk menganalisis semantik suara pengguna secara cerdas untuk mengekstraksi target tujuan (air panas, biasa, atau dingin). Pada sisi persepsi visual, stasiun kamera depth Intel RealSense D435i melokalisasi objek target menggunakan model YOLO kustom, sementara MediaPipe mengekstraksi koordinat 3 dimensi dari lima titik ujung jari pengguna secara berkelanjutan. Vektor selisih jarak spasial dikomputasi secara real-time dan ditransmisikan melalui protokol UDP menuju mikrokontroler ESP32 pada sarung tangan haptik. Umpan balik taktil diberikan melalui aktuasi empat motor vibrasi koin yang memandu arah gerak tangan menggunakan paradigma dorong-tarik (push-pull). Hasil pengujian tahap awal menunjukkan bahwa kalibrasi offset koordinat berhasil mengatasi distorsi paralaks lensa, dan sistem secara keseluruhan mampu memberikan panduan haptik yang responsif untuk menuntun tangan pengguna menuju koordinat target dengan aman. Meskipun demikian, evaluasi menunjukkan bahwa aktuasi vertikal (naik atau turun) yang menggunakan kombinasi dua motor secara bersamaan masih memberikan beban kognitif tambahan bagi pengguna.
===================================================================================================================================
Visually impaired users often face spatial orientation challenges when interacting with modern household appliances. One of the main challenges lies in operating under-sink water dispensers, the majority of which feature a flat interface without tactile markers, thereby posing safety risks such as accidentally pressing the hot water button. This research proposes the design of a Vision System as a touchless spatial navigation aid that integrates natural language processing, 3D computer vision, and wearable actuation. The system utilises a Speech-to-Text interface and a Large Language Model (LLM) to intelligently analyse the semantic meaning of the user’s voice in order to extract the target selection (hot, normal, or cold water). On the visual perception side, an Intel RealSense D435i depth camera localises the target object using a custom YOLO model, whilst MediaPipe continuously extracts the 3D coordinates of the user’s five fingertips. Spatial distance difference vectors are computed in real time and transmitted via the UDP protocol to the ESP32 microcontroller in the haptic glove. Tactile feedback is provided via the actuation of four coin-sized vibration motors, which guide the direction of hand movement using a push-pull paradigm. Preliminary test results indicate that coordinate offset calibration successfully compensates for lens parallax distortion, and the system as a whole is capable of providing responsive haptic guidance to safely guide the user’s hand towards the target coordinates. However, the evaluation suggests that vertical actuation (up or down) utilising a combination of two motors simultaneously still imposes an additional cognitive load on the user.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Large Language Model (LLM), Sistem Asistif, Tunanetra, Umpan Balik Taktil, Visi Komputer, 3D Hand Tracking
Subjects:	Q Science > QA Mathematics > QA336 Artificial Intelligence R Medicine > R Medicine (General) > R858 Deep Learning T Technology > T Technology (General) > T59.7 Human-machine systems. T Technology > TJ Mechanical engineering and machinery > TJ211 Robotics.
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Electrical Engineering > 20201-(S1) Undergraduate Thesis
Depositing User:	Aloysius Abraham Silalahi
Date Deposited:	27 Jul 2026 07:04
Last Modified:	27 Jul 2026 07:04
URI:	http://repository.its.ac.id/id/eprint/138003

Actions (login required)

View Item