A Workload-Aware Hybrid Cache Eviction and Asynchronous Prefetching Strategy for Tail Latency Reduction in Real-Time Streaming Data Pipelines

Abstract

Real-time streaming data pipelines that feed downstream model inference have become a critical infrastructure component for latency-sensitive analytics and online inference, including operational monitoring, ad-tech serving, and other applications requiring predictable, low-latency processing. While average end-to-end latency in modern stream engines has reached the millisecond level, the 99th-percentile (P99) tail latency often remains an order of magnitude higher than the median and can violate service-level objectives during load spikes and stateful operator stalls. This paper studies the joint optimisation of cache eviction and asynchronous prefetching at the algorithmic layer, without modifying the underlying engine or the inference model. We propose WHEAP (Workload-aware Hybrid Eviction with Asynchronous Prefetching), a strategy that adaptively reweights frequency- and recency-based eviction signals according to online workload skew, and that decouples the data path from state access through key-aware prefetching with a bounded in-flight window. We evaluate WHEAP on three public streaming benchmarks (NEXMark, the Yahoo Streaming Benchmark, and ShuffleBench) for stateful queries with state sizes ranging from 1.2 GB to 4.8 GB and event rates up to 200,000 events per second. Compared with LRU, ARC, and Async+LRU baselines (with LFU also reported on a representative query), WHEAP reduces P99 latency by 18.4% to 27.3% in our experiments while keeping cache hit-rate degradation under 1.6 percentage points and maintaining throughput within 4% of the strongest baseline. Each configuration is repeated three times, and we report the median together with the inter-run spread to characterise statistical variability.