Studi Komparatif Base Model, Tool-Augmented, Dan Ai Agent Pada Konseling Kesehatan Mental Menggunakan Model QWEN

Teja, Andika Rahman (2026) Studi Komparatif Base Model, Tool-Augmented, Dan Ai Agent Pada Konseling Kesehatan Mental Menggunakan Model QWEN. Other thesis, Institut Teknologi Sepuluh Nopember.

Text 5025221022-Undergraduate_Thesis.pdf - Accepted Version Restricted to Repository staff only Download (8MB) | Request a copy

Abstract

Masalah kesehatan mental yang dialami oleh remaja di Indonesia dipengaruhi oleh berbagai faktor. Salah satunya adalah keterbatasan akses layanan psikologis yang terjangkau dan mudah diakses. Penelitian ini bertujuan untuk memberikan solusi dalam bentuk sistem digital berbasis AI agent dengan memanfaatkan model Qwen yang memiliki kemampuan reasoning dan penggunaan alat eksternal secara open-source. Selain menggunakan pendekatan AI agent, penelitian ini juga membandingkan baseline model, model fine-tuned (Emilia), dan pendekatan tool-augmented LLM. Evaluasi dilakukan menggunakan data CounselBench untuk interaksi tunggal dan data MindEval untuk interaksi berkelanjutan dengan menggunakan evaluator ahli psikologi dan LLM-as-a-Judge. Validasi reliabilitas menunjukkan kesepakatan fair antara psikolog dan LLM-as-a-Judge pada data CounselBench (QWK pooled = 0,399), namun tidak berkorelasi pada data MindEval (QWK pooled = 0,010) sehingga uji signifikansi data CounselBench menggunakan penilaian LLM-as-a-Judge dan data MindEval menggunakan penilaian psikolog. Hasil uji Friedman menunjukkan seluruh metrik CounselBench berbeda secara signifikan (p < 0,001), dengan Qwen3.5 (Base) meraih skor rerata tertinggi (M = 3,983). Selain itu, model Emilia menunjukkan keunggulan pada aspek Naturalness, Dialogic, dan Prescriptive, namun mengalami trade-off pada aspek Empathy. Pada data MindEval, tidak ditemukan perbedaan signifikan antarkonfigurasi akibat keterbatasan jumlah blok data.
=================================================================================================================================
Mental health problems experienced by adolescents in Indonesia are influenced by various factors, one of which is the limited access to affordable and accessible psychological services. This study aims to provide a solution in the form of a digital system based on an AI agent by leveraging the Qwen model, which possesses open-source reasoning capabilities and external tool usage. In addition to the AI agent approach, this study also compares a baseline model, a fine-tuned model (Emilia), and a tool-augmented LLM approach. Evaluation was conducted using the CounselBench dataset for single-turn interactions and the MindEval dataset for multi-turn interactions, employing both expert psychologists and an LLM-as-a-Judge as evaluators. Reliability validation indicated fair agreement between psychologists and LLM-as-a-Judge on the CounselBench data (pooled QWK = 0.399), but no meaningful correlation on the MindEval data (pooled QWK = 0.010) consequently, significance testing for CounselBench relied on LLM-as-a-Judge ratings, while MindEval relied on psychologist ratings. Friedman test results showed that all CounselBench metrics differed significantly (p < 0.001), with Qwen3.5 (Base) achieving the highest mean score (M = 3.983). Furthermore, the Emilia model demonstrated superiority in the Naturalness, Dialogic, and Prescriptive dimensions, yet exhibited a trade-off in the Empathy dimension. On the MindEval data, no significant differences were found among configurations due to the limited number of data blocks.

Item Type:	Thesis (Other)
Uncontrolled Keywords:	Kesehatan Mental, Mental Health, Large Language Model, Qwen, Psikolog, Psychologist, LLM-as-a-Judge
Subjects:	Q Science > QA Mathematics > QA336 Artificial Intelligence R Medicine > R Medicine (General) > R858 Deep Learning
Divisions:	Faculty of Intelligent Electrical and Informatics Technology (ELECTICS) > Informatics Engineering > 55201-(S1) Undergraduate Thesis
Depositing User:	Andika Rahman Teja
Date Deposited:	19 Jun 2026 00:34
Last Modified:	19 Jun 2026 00:34
URI:	http://repository.its.ac.id/id/eprint/133912

Actions (login required)

View Item