Application of Anomaly Detection Mechanism in Large-Scale Data Processing

Yixin Zhou

Authors

Yixin Zhou Amazon, Ads API Infra, New York, 10001, USA Author

Keywords:

anomaly detection mechanism, large scale data processing, financial monitoring, equipment failure prediction, network security

Abstract

Anomaly detection technology plays a crucial role in large-scale data processing and is widely used in multiple industries such as finance, the industrial Internet of Things, information security, and intelligent transportation systems such as finance, industrial Internet of Things, information security, and intelligent transportation systems. This technology is dedicated to discovering abnormal behaviors or patterns in large and complex datasets, with the aim of enhancing the accuracy and reliability of the data processing process. This article explores the specific applications of anomaly detection in abnormal transaction detection in the financial industry, device failure prediction in industrial Internet of Things, intrusion detection in network security, and abnormal traffic monitoring in smart transportation. It demonstrates the important role of this mechanism in optimizing business processes, strengthening security, and enhancing risk management capabilities. The trend of intelligent data processing is driven by anomaly detection technology, which significantly improves the ability to process large amounts of data and provides solid technological support for data-driven decision-making and management in various industries.

References

1. W. Liu, L. Yan, N. Ma, G. Wang, X. Ma, P. Liu, and R. Tang, “Unsupervised deep anomaly detection for industrial multivariate time series data,” Appl. Sci., vol. 14, no. 2, p. 774, 2024, doi: 10.3390/app14020774.

2. Z. Wang, C. Pei, M. Ma, X. Wang, Z. Li, D. Pei, and G. Xie, “Revisiting VAE for unsupervised time series anomaly detection: A frequency perspective,” in Proc. ACM Web Conf. (WWW), 2024, pp. 3096–3105, doi: 10.1145/3589334.3645710.

3. D. K. Thakur, K. P. Sivaraj, M. Gulhane, J. Alahari, R. Jena, and P. Goel, “Efficient network anomaly detection and mitigation strategies for large-scale networks,” in Proc. 2025 Int. Conf. Pervasive Comput. Technol. (ICPCT), 2025, pp. 279–283, doi: 10.1109/ICPCT64145.2025.10940544.

4. H. Luo, Y. Zheng, K. Chen, and S. Zhao, “Probabilistic temporal fusion transformers for large-scale KPI anomaly detection,” IEEE Access, vol. 12, pp. 9123–9137, 2024, doi: 10.1109/ACCESS.2024.3353201.

5. Z. Jia, Z. Wang, Z. Sun, X. Sun, P. Liu, and F. Ruzzenenti, “A multi-scenario data-driven approach for anomaly detection in electric vehicle battery systems,” eTransp., vol. 24, p. 100418, 2025, doi: 10.1016/j.etran.2025.100418.