iKafka: Intelligent Storage Management for Adaptive Event Streaming in Kafka

Abstract

Continuous evaluation of Distributed Computing Continuum Systems heavily relies on efficient communication models, with event streaming Publish-Subscribe (Pub-Sub) systems playing a key role in ensuring fault tolerance, scalability, and real-time analytics across heterogeneous tiers. On event streaming platforms like Apache Kafka, consumer applications often exhibit periodic or event-driven patterns when revisiting historical events, which requires storing these events over time. However, accumulating all events increases storage demands. To address this challenge, Kafka's default log retention policies, governed by static time or size thresholds, may prematurely delete data that will be revisited in future cycles. Unfortunately, static time- or size-based retention policies are insufficient, as they fail to maintain equilibrium between resource utilization, cost, and quality of service (QoS). As a result, intelligent and adaptive storage management strategies are required to minimize storage requirements while maintaining enhanced QoS. In this context, we propose a Light Gradient Boosting Machine (LightGBM)-based adaptive storage optimization in event streaming Kafka broker (namely, iKafka) to identify periodic consumer patterns and determine near-optimal retention times for events. iKafka also considers adversarial attacks by monitoring prediction accuracy to determine whether to use the predicted retention times or revert to default retention times. We evaluate the proposed iKafka system with an air quality use case, and our results demonstrate approximately 5.5× less memory resources over traditional Kafka under ideal conditions.