Distributed Index Optimization Techniques for Time-Sensitive Massive Data

Abstract

With the continuous deployment of industrial perception networks and smart urban monitoring systems, the volume of temporal stream data has expanded significantly, evolving from terabyte-level to petabyte-scale volumes. The high-throughput and persistent nature of data generation imposes stringent demands on storage and retrieval architectures. Traditional single-machine indexing architectures suffer from limited computational power and storage capacity, making them inadequate for processing ultra-large-scale temporal datasets. Existing distributed indexing solutions predominantly employ fixed sharding mechanisms, often encountering challenges such as uneven node load distribution, high query interaction costs across nodes, and inefficient reuse of cold and hot data indices, thereby failing to simultaneously ensure stable high-concurrency writes and low-latency multi-dimensional queries. Addressing these technical challenges, this paper proposes a dual-layer hybrid distributed indexing architecture that integrates temporal data characteristics with hierarchical storage techniques. The proposed approach compresses index metadata using temporal feature hash summaries, implements dynamic scheduling and partition pruning mechanisms based on data access frequency, and optimizes the underlying logic for log-structured merge tree merging operations. Experimental results across multiple benchmark datasets confirm the effectiveness of the proposed solution in reducing indexing overhead, substantially lowering cross-partition query latency, and achieving balanced cluster workload distribution. The findings provide a viable and scalable technical framework for the efficient distributed retrieval of massive temporal datasets in real-world industrial and urban monitoring applications.