Integrasi Home Assistant Berbasis Pengenalan Wajah Dan Perintah Suara

Musyaffa, Ahnaf (2025) Integrasi Home Assistant Berbasis Pengenalan Wajah Dan Perintah Suara. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5027211038-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (9MB) | Request a copy

Abstract

Perkembangan teknologi Internet of Things (IoT) dan Artificial Intelligence (AI) membuka peluang besar dalam menciptakan solusi cerdas di berbagai lingkungan, termasuk kampus. Home Assistant merupakan platform otomasi yang banyak digunakan, namun masih memiliki keterbatasan pada fitur berbasis AI. Penelitian ini bertujuan mengembangkan sistem otomasi berbasis pengenalan wajah dan perintah suara dengan mengintegrasikan computer vision dan large language model dengan platform Home Assistant. Sistem deteksi dan pengenalan wajah dibangun menggunakan YOLOv11n dan FaceNet512, sedangkan perintah suara diproses menggunakan Whisper untuk transkripsi audio dan LLM Phi-4 mini untuk intent matching dan person information retrieval. Sistem mencatat keberhasilan 80% dalam menghasilkan respons sesuai perintah bahasa Indonesia dan Inggris pada sistem intent matching, serta keberhasilan 90% (bahasa Indonesia) dan 100% (bahasa Inggris) pada sistem person information retrieval.Sistem diakses melalui webapp berbasis Flask yang mendukung input wajah dan suara secara langsung, serta terhubung ke platform Home Assistant melalui protokol MQTT. Hasil pengujian menunjukkan akurasi 100% dalam skenario variasi jarak, pencahayaan, dan penggunaan kacamata, serta akurasi 100% pada jarak 200–300 cm untuk kategori dosen dan tenaga kependidikan. Pada skenario satu wajah, sistem mencapai presisi 83,8% dan F1-score 0,912, namun menurun menjadi 63,6% dan F1-score 0,448 dalam pengenalan banyak wajah. Sistem perintah suara secara konsisten menunjukkan performa tinggi dalam memahami maksud dan mengambil informasi pengguna.
===================================================================================================================================
The development of Internet of Things (IoT) and Artificial Intelligence (AI) technologies opens up significant opportunities for creating intelligent solutions in various environments, including campuses. Home Assistant is a widely used automation platform, but it still has limitations in AI-based features. This study aims to develop an automation system based on facial recognition and voice commands by integrating computer vision and large language models with Home Assistant. The face detection and recognition system is built using YOLOv11n and FaceNet512, while voice commands are processed using Whisper for audio transcription and the Phi-4 mini LLM for intent matching and person information retrieval. The system achieved an 80% success rate in producing appropriate responses to voice commands in both Indonesian and English for the intent matching task, and 90% (Indonesian) and 100% (English) success for person information retrieval. The system is accessed via a Flask-based webapp that supports real-time facial and voice input and is integrated with Home Assistant via the MQTT protocol. Test results show 100% accuracy under varying conditions of distance, lighting, and eyeglass usage, as well as 100% accuracy at a distance of 200–300 cm for the lecturer and staff category. In single-face scenarios, the system achieved 83.8% accuracy and an F1-score of 0.912, but performance dropped to 63.6% accuracy and an F1-score of 0.448 in multi-face recognition. The voice command system consistently demonstrated high performance in understanding user intent and retrieving user information.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	FaceNet512, Integrasi Home Assistant, Pengenalan Wajah, Perintah Suara, YOLO, Face Recognition, FaceNet512, Home Assistant Integration, Voice Command, YOLO
Subjects:	T Technology > T Technology (General) > T58.8 Productivity. Efficiency T Technology > T Technology (General) > T59.7 Human-machine systems.
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Information Technology > 59201-(S1) Undergraduate Thesis
Depositing User:	Ahnaf Musyaffa
Date Deposited:	21 Jul 2025 01:16
Last Modified:	21 Jul 2025 01:16
URI:	http://repository.its.ac.id/id/eprint/120150

Actions (login required)

View Item