Alamsyah, Muhamad Rizano (2026) Sistem Inspeksi Visual pada Robot Quadruped berbasis Multimodal Large Language Model dengan Mekanisme Human-in-the-Loop. Other thesis, Institut Teknologi Sepuluh Nopember.
![[thumbnail of 5024221014-Undergraduate_Thesis.pdf]](http://repository.its.ac.id/style/images/fileicons/text.png) |
Text
5024221014-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (14MB) | Request a copy |
Abstract
Penelitian ini bertujuan mengintegrasikan agen Multimodal Large Language Model (MLLM) dengan robot Jueying Lite3 untuk mengotomatisasi navigasi dan analisis visual, sekaligus merancang arsitektur Human-in-the-Loop (HITL) untuk menjamin adaptabilitas sistem di tengah misi inspeksi. Pengembangan sistem menggunakan standar Model Context Protocol (MCP) dan webhook untuk menghubungkan instruksi bahasa alami dengan navigasi ROS Noetic pada robot. Instruksi dipecah menjadi sub-tugas sekuensial yang memerlukan validasi operator di tiap titik koordinat. Pada setiap titik inspeksi, robot menangkap citra objek dan membandingkannya dengan dokumen Standard Operating Procedure (SOP) untuk menilai kesesuaian kondisi. Evaluasi dilakukan melalui pengujian komponen secara terisolasi (pemanggilan tools, analisis image, pembuatan plan), pengujian keseluruhan sistem secara end-to-end, dan pengukuran beban kerja operator menggunakan kuesioner NASA-TLX. Pada pengujian terisolasi, Gemini 3.1 Pro mencatatkan keberhasilan pemanggilan tools 98,61% (71/72), akurasi analisis visual 100% (27/27), dan efisiensi rute plan 100% (12/12). Pada pengujian end-to-end terhadap 9 skenario misi bertingkat (mudah, sedang, sulit), Gemini 3.1 Pro mencatatkan akurasi analisis visual 93,33% (28/30 titik inspeksi) dan efisiensi rute 100% (9/9 skenario). Sistem juga berhasil mengakomodasi seluruh 4 skenario intervensi operator yang diujikan (perubahan rencana awal, penambahan objek di tengah misi, pembatalan dan penggantian objek, serta gerakan tambahan manual), serta menurunkan total skor beban kerja NASA-TLX dari 54,13 (pada teleoperasi) menjadi 18,40. Arsitektur MLLM dengan mekanisme HITL mampu menangani skenario inspeksi yang berbeda, baik dari segi perbedaan rute maupun variasi objek target, secara adaptif tanpa perubahan kode, mampu membuat sistem beradaptasi di tengah berjalannya misi dan berhasil meringankan beban kognitif operator secara signifikan dalam inspeksi otonom.
================================================================================================================================
This research aims to integrate a Multimodal Large Language Model (MLLM) agent with the Jueying Lite3 robot to automate navigation and visual analysis, while designing a Human-in-the-Loop (HITL) architecture to ensure system adaptability mid-inspection mission. The system development utilizes the Model Context Protocol (MCP) standard and webhooks to connect natural language instructions with ROS Noetic navigation on the robot. Instructions are broken down into sequential sub-tasks requiring operator validation at each coordinate. At every inspection point, the robot captures an image of the object and compares it against Standard Operating Procedure (SOP) documents to assess condition compliance. Evaluation was conducted through isolated component testing (tool calling, image analysis, plan generation), end-to-end system testing, and operator workload measurement using the NASA-TLX questionnaire. In isolated testing, Gemini 3.1 Pro achieved a tool calling success rate of 98.61% (71/72), visual analysis accuracy of 100% (27/27), and plan route efficiency of 100% (12/12). In end-to-end testing across 9 tiered mission scenarios (easy, medium, hard), Gemini 3.1 Pro recorded a visual analysis accuracy of 93.33% (28/30 inspection points) and a route efficiency of 100% (9/9 scenarios). The system also successfully accommodated all 4 tested operator intervention scenarios (initial plan alteration, mid-mission object addition, object cancellation and replacement, and manual secondary movement), while reducing the total NASA-TLX workload score from 54.13 (under teleoperation) to 18.40. The MLLM architecture coupled with the HITL mechanism is capable of adaptively handling varying inspection scenarios in terms of route differences and target object variations without code modifications, ensuring system adaptability mid-mission, and significantly alleviating operator cognitive load in autonomous inspections.
Actions (login required)
 |
View Item |