Deteksi Bagian Funsional pada Peralatan Dapur Menggunakan Segmentasi Berbasis Penalaran

Nurhasan, Arundaya Pratama (2026) Deteksi Bagian Funsional pada Peralatan Dapur Menggunakan Segmentasi Berbasis Penalaran. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5025221206-Undergraduate_Thesis.pdf Restricted to Repository staff only Download (7MB) | Request a copy

Abstract

Penyerahan objek (handover) menjadi salah satu tugas umum dalam interaksi manusia dan robot di lingkungan rumah, namun robot masih kesulitan mengenali bagian mana yang aman digenggam dan bagian mana yang berisiko melukai penerima, terutama pada benda tajam seperti pisau atau gunting. Model segmentasi berbasis neural network seperti U-Net dan YOLO-seg tidak mampu memahami instruksi bahasa, sementara model transformer seperti X-Decoder dan OVSeg baru bisa mengenali objek secara utuh tanpa membedakan bagian fungsionalnya. Di sisi lain, model berbasis LLM seperti LISA dan PISA hanya bekerja baik pada instruksi eksplisit dan gagal menangkap maksud tersirat dari perintah sehari-hari. Kombinasi keterbatasan ini, ditambah mahalnya biaya pembuatan dataset spesifik secara manual, menjadi celah (gap) yang mendasari penelitian ini. Untuk mengatasi masalah tersebut, penelitian ini mengembangkan PACT-Seg (Part Affordance and Contextual Task Segmentation), yang menghubungkan LLaVA dan SAM melalui token [SEG] sehingga pemahaman bahasa dari LLaVA dapat diteruskan menjadi masker segmentasi oleh SAM. Model dilatih menggunakan dua jenis dataset yang saling melengkapi. Jenis pertama berbasis gambar sintetis hasil rendering objek 3D Objaverse LVIS dengan domain randomization, yang masker segmentasinya dihasilkan secara otomatis menggunakan SAM3. Jenis kedua berbasis gambar real-world dari dataset InstructPart dan PACO, yang masker segmentasinya berasal dari anotasi manual yang telah tersedia pada kedua dataset tersebut. Pada kedua jenis dataset, variasi instruksi (implicit, explicit, dan region-aware) dalam Bahasa Inggris dan Bahasa Indonesia disintesis menggunakan LLM Qwen. Performa model diuji melalui tiga skema evaluasi, yaitu perbandingan instruksi implisit-eksplisit-region-aware, perbandingan bahasa Inggris-Indonesia, serta perbandingan citra sintesis dengan foto dunia nyata, dengan metrik gIoU, cIoU, Dice Score, dan P@50-95. Hasil pengujian menunjukkan model usulan (PACT-Seg) berhasil mengungguli model baseline LISA maupun model segmentasi umum SAM3 pada seluruh skema pengujian. Pada pengujian menggunakan citra sintesis, PACT-Seg mencapai gIoU sebesar 0,8727 dan Dice Score 0,9171, jauh di atas baseline yang hanya mencapai gIoU 0,3766. Model usulan juga menunjukkan performa yang stabil pada kisaran gIoU 0,77 baik untuk instruksi implicit, explicit, maupun region-aware, serta tetap stabil (gIoU 0,7680 menjadi 0,7705) ketika instruksi diberikan dalam Bahasa Indonesia dibandingkan Bahasa Inggris, berbeda dengan baseline dan SAM3 yang performanya turun tajam pada kedua kondisi tersebut. Performa paling optimal diperoleh ketika PACT-Seg dilatih dan diuji menggunakan seluruh dataset gabungan (gambar sintetis dan real-world) dengan kedua bahasa sekaligus, yaitu gIoU sebesar 0,7705 dan Dice Score 0,8285 pada Bahasa Indonesia, serta gIoU 0,7680 dan Dice Score 0,8261 pada Bahasa Inggris. Namun, ketika diuji pada foto dunia nyata, performa model menurun cukup tajam dari gIoU 0,8727 menjadi 0,6398 akibat sim-to-real gap, dan nilai P@50-95 yang turun lebih tajam (38,1%) dibandingkan Dice Score (22,1%) menunjukkan model masih kesulitan menghasilkan kontur masker yang presisi meski area umumnya sudah dipahami dengan benar. Penelitian ini menyimpulkan bahwa pipeline pembuatan dataset otomatis berhasil menutup reasoning gap pada instruksi implisit dan multibahasa, tetapi teknik domain randomization standar belum cukup untuk mengatasi sim-to-real gap sepenuhnya, sehingga diperlukan pendekatan tambahan pada penelitian selanjutnya.
================================================================================================================================
Object handover is one of the common tasks in human-robot interaction within household environments, yet robots still struggle to recognize which part of an object is safe to grasp and which part poses a risk of injuring the recipient, particularly for sharp objects such as knives or scissors. Neural-network-based segmentation models such as U-Net and YOLO-seg are unable to understand language instructions, while transformer-based models such as X-Decoder and OVSeg can only recognize objects as a whole without distinguishing their functional parts. Meanwhile, LLM-based models such as LISA and PISA only work well on explicit instructions and fail to capture the implicit intent of everyday commands. This combination of limitations, together with the high cost of manually building task-specific datasets, forms the gap that motivates this research. To address this problem, this research develops PACT-Seg (Part Affordance and Contextual Task Segmentation), which connects LLaVA and SAM through a [SEG] token so that the language understanding produced by LLaVA can be passed on as a segmentation mask by SAM. The model is trained using two complementary types of datasets. The first is based on synthetic images rendered from Objaverse LVIS 3D assets with domain randomization, whose segmentation masks are generated automatically using SAM3. The second is based on real-world images from the InstructPart and PACO datasets, whose segmentation masks come from the manual annotations already available in both datasets. For both dataset types, instruction variations (implicit, explicit, and region-aware) in English and Indonesian are synthesized using the Qwen LLM. Model performance is tested through three evaluation schemes, namely implicit-explicit-region-aware instruction comparison, English-Indonesian language comparison, and synthetic-real-world image comparison, using the gIoU, cIoU, Dice Score, and P@50-95 metrics. The test results show that the proposed model (PACT-Seg) successfully outperforms both the LISA baseline and the general-purpose SAM3 segmentation model across all evaluation schemes. On synthetic images, PACT-Seg achieves a gIoU of 0.8727 and a Dice Score of 0.9171, far above the baseline's gIoU of 0.3766. The proposed model also maintains stable performance around a gIoU of 0.77 across implicit, explicit, and region-aware instructions, and remains stable (gIoU 0.7680 to 0.7705) when instructions are given in Indonesian rather than English, unlike the baseline and SAM3, whose performance drops sharply under both conditions. The most optimal performance is obtained when PACT-Seg is trained and tested using the entire combined dataset (synthetic and real-world images) with both languages at once, reaching a gIoU of 0.7705 and a Dice Score of 0.8285 in Indonesian, and a gIoU of 0.7680 and a Dice Score of 0.8261 in English. However, when tested on real-world photos, model performance drops considerably from a gIoU of 0.8727 to 0.6398 due to the sim-to-real gap, and a P@50-95 score that drops more sharply (38.1%) than the Dice Score (22.1%) indicates that the model still struggles to produce precise mask contours even though the general area is usually understood correctly. This research concludes that the automatic dataset generation pipeline successfully closes the reasoning gap for implicit and multilingual instructions, but the standard domain randomization technique is not yet sufficient to fully overcome the sim-to-real gap, so additional approaches are needed in future research.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	InteraksiManusia-Robot,PenyerahanBarang,Affordance,Reasoning Segmentation,AnotasiOtomatis,LoRA. Human-Robot Interaction,ObjectHandover,Affordance,Reasoning Segmentation,AutomaticAnnotation,LoRA.
Subjects:	A General Works > AI Indexes (General) A General Works > AI Indexes (General) Q Science > QA Mathematics > QA76.87 Neural networks (Computer Science)
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering
Depositing User:	Arundaya Pratama Nurhasan
Date Deposited:	30 Jul 2026 08:26
Last Modified:	30 Jul 2026 08:26
URI:	http://repository.its.ac.id/id/eprint/139517

Actions (login required)

View Item