Implementasi Sistem Antrean Berbasis Message Broker Menggunakan Celery untuk Integrasi Data Terdistribusi pada Pipeline ETL

Ramadhani, Azka Rizqullah (2025) Implementasi Sistem Antrean Berbasis Message Broker Menggunakan Celery untuk Integrasi Data Terdistribusi pada Pipeline ETL. Project Report. [s.n.], [s.l.]. (Unpublished)

Text 5025231148-Project_Report.pdf - Accepted Version Restricted to Repository staff only Download (1MB) | Request a copy

Abstract

Penelitian ini bertujuan untuk mengimplementasikan sistem integrasi data terdistribusi pada pipeline ETL (Extract, Transform, Load) menggunakan sistem antrean berbasis message broker. Permasalahan utama dalam integrasi data adalah adanya beban kerja yang berat dan risiko kegagalan sinkronisasi saat menarik data dari berbagai sumber secara bersamaan. Untuk mengatasi hal tersebut, penelitian ini menerapkan pustaka Celery dengan message broker (Redis) untuk mengelola tugas secara asinkron. Data diambil dari tiga sumber berbeda, yaitu REST API, GraphQL, dan PostgreSQL. Kemudian data akan di transformasi dan dibersihkan sebelum memasuki Gudang penyimpanan. Hasil pengujian menunjukkan bahwa arsitektur antrean berhasil menjamin reliabilitas, mencegah duplikasi, dan menjalankan seluruh proses secara otomatis melalui Celery Beat. Penelitian ini menyimpulkan bahwa penggunaan Celery dan Redis dapat meningkatkan efisiensi dan keandalan sistem ETL.
====================================================================================================================================
This research aims to implement a distributed data integration system within an ETL (Extract, Transform, Load) pipeline using a message broker-based queueing system. The primary challenges in data integration are heavy workloads and the risk of synchronization failures when extracting data simultaneously from multiple sources. To address these issues, this study utilizes the Celery library with a message broker (Redis) to manage tasks asynchronously. Data is extracted from three distinct sources: REST API, GraphQL, and PostgreSQL. Subsequently, the data is transformed and cleaned before entering the data warehouse. Test results demonstrate that the queue architecture successfully ensures reliability, prevents duplication, and executes the entire process automatically via Celery Beat. This study concludes that the use of Celery and Redis can enhance the efficiency and reliability of the ETL system.

Item Type:	Monograph (Project Report)
Uncontrolled Keywords:	Pipeline ETL, Celery, Message Broker, Redis
Subjects:	T Technology > T Technology (General) > T57.5 Data Processing
Divisions:	Faculty of Industrial Technology > Informatics Engineering > 55201-(S1) Undergraduate Thesis
Depositing User:	Azka Rizqullah Ramadhani
Date Deposited:	30 Dec 2025 04:23
Last Modified:	30 Dec 2025 04:23
URI:	http://repository.its.ac.id/id/eprint/129174

Actions (login required)

View Item