Lightweight LSTM-Based Adaptive Kafka Tuning for Predictive IoT Data Streams

Abstract

The Internet of Things (IoT) is a fundamental element of the computing continuum, characterized by data streams that exhibit predictable patterns primarily driven by sensor configurations and deployment strategies. On the other hand, Apache Kafka, renowned for its high-throughput and faulttolerant data streaming capabilities, is well-suited for managing IoT data streams. However, static Kafka configurations often result in inefficiencies such as suboptimal batching, increased consumer lag, and underutilization of system resources. To address these challenges, we propose a dynamic reconfiguration approach that leverages short-term historical data to forecast message rates and adjust Kafka parameters in real time using a lightweight Long Short-Term Memory (LSTM) model. This adaptive approach optimizes the configuration of Kafka producers and consumers for IoT environments, achieving a prediction accuracy of 91.42% with minimal computational overhead. Experimental evaluations demonstrate substantial improvements in consumer lag reduction, throughput stability, and CPU utilization across heterogeneous IoT workloads, with the system requiring only brief observation periods to effectively tune performance.