ADAPTATION OF THE ALGORITHM FOR DETECTING ANOMALIES IN TIME SERIES FOR NON-STATIONARY STREAMING DATA
DOI:
https://doi.org/10.33042/2522-1809-2024-3-184-16-22Keywords:
concept drift, extreme value theory, multivariate time series, outlier detectionAbstract
Detection of anomalies in streaming time series data has become an important research topic due to the wide range of possible applications, including the detection of extreme weather conditions, malicious attacks on protected facilities, monitoring unauthorised gas and oil leaks, illegal pipeline connections, power cable faults, and water and other environmental pollution. Rapid detection of abnormal conditions and identifying these critical events is essential to protect lives and assets. Therefore, developing appropriate systems and detection methods is an urgent task.
Anomaly detection in streaming data is challenging due to its large volume and high speed, presence of noise, and non-stationarity of the signal (or ‘concept drift’). The latter significantly complicates the identification of differences between new ‘typical’ behaviour and abnormal events.
Solving this problem requires the algorithm for processing such data to learn and adapt to changing conditions.
The paper proposes a modification of the algorithm for detecting anomalies in time series. This algorithm provides early detection of abnormal series in a massive collection of non-stationary streaming time series data. Anomalies are observations that are highly improbable given the previous time series values. The proposed approach is based on the primary detection of the predictive limit for the typical system behaviour using the theory of extreme values, followed by checking for the abnormality of the following series using the sliding window technique. The time series parameters are used as input data and compared by density distribution to detect any significant changes in the distribution of the characteristics. It allows the decision-making model to automatically adapt to the changing environment according to detected changes.
Since anomalies are, by definition, exceptions to the typical behaviour of a system, most of the available stored data should reflect that typical behaviour of the system in question. It is unnecessary to have representative samples of all possible types of the standard behaviour of a given system for the algorithm to work well. The basic idea is to have a warm-up data set to obtain initial values for the decision model parameters. It makes it possible to determine if there is any significant difference between the last typical behaviour and the new typical behaviour.
The proposed algorithm demonstrates its performance under conditions of noisy non-stationary data in several time series classes.
References
Lavin A. Evaluating Real-Time Anomaly Detection Algorithms–The Numenta Anomaly Benchmark / Lavin A, Ahmad S. // Machine Learning and Applications (ICMLA), IEEE 14th International Conference on. IEEE, 2015, pp. 38–44.
Russakovsky O. Imagenet large scale visual recognition challenge / Russakovsky O // International journal of computer vision 115.3, 2015: 211–252.
Makridakis S. The M5 accuracy competition: Results, findings and conclusions / S.Makridakis, E. Spiliotis, V. As-simakopoulos // Int J Forecast, 2020.
Makridakis S. The M5 accuracy competition: Results, findings and conclusions / S.Makridakis, E. Spiliotis, V. As-simakopoulos // International Journal of Forecasting (2020): 1–24.
Tziolas, T., Papageorgiou, K., Theodosiou, T., Papageorgiou, E., Mastos, T., & Papadopoulos, A. (2022). Autoencoders for Anomaly Detection in an Industrial Multivariate Time
Series Dataset. Engineering Proceedings, 18(1), 23. https://doi.org/10.3390/engproc2022018023
Ruff L. Deep semi-supervised anomaly detection / Ruff L. // arXiv preprint arXiv:1906.02694, - 2019.
Krohn D. Fiber optic sensors: fundamentals and applications / D.A Krohn, T. MacDougall, A. Mendez // Isa, - 2000.
Catalano A. An intrusion detection system for the protection of railway assets using Fiber Bragg Grating sensors / A Cata-lano, FA Bruno, M Pisco, A Cutolo, A Cusano // Sensors 14(10), 2014, 18268–18285.
Gupta M. Outlier detection for temporal data: A survey / M Gupta, J Gao, C Aggarwal, J Han // IEEE Transactions on Knowledge and Data Engineering 26(9), 2014, 2250–2267.
Hayes M. Contextual anomaly detection framework for big sensor data / M. Hayes, M. Capretz // Journal of Big Data 2(1), 2015.
Schwarz K. Wind dispersion of carbon dioxide leaking from underground sequestration, and outlier detection in eddy covariance data using extreme value theory / K. Schwarz // ProQuest. 2008.
Burridge P. Additive Outlier Detection Via Extreme-Value Theory / P. Burridge, A. Taylor // Journal of Time Series Analysis 27(5), 2066, р. 685–701.
Hyndman RJ. Large-scale unusual time series detection. In: Data Mining Workshop (ICDMW)/ RJ Hyndman, E Wang, N Laptev // IEEE International Conference on. IEEE, - 2015, pp. 1616–1619.
Wilkinson L. Visualizing Big Data Outliers through Dis-tributed Aggregation / L. Wilkinson // IEEE transactions on visualization and computer graphics 24(1), 2018, р. 256–266.
Perron P. Searching for additive outliers in nonstationary time series / P. Perron, G Rodríguez // Journal of Time Series Analysis 24(2), 2009, р. 193–220.
Sundaram S. Aircraft engine health monitoring using density modelling and extreme value statistics. / Sundaram S, IGD Strachan, DA Clifton, L Tarassenko, S King // Proceed-ings of the 6th International Conference on Condition Moni-toring and Machine Failure Prevention Technologies, 2009.
S. Hugueny. Novelty detection with extreme value theory in vital-sign monitoring. PhD thesis. University of Oxford, 2013.
Fulcher BD. Highly comparative time-series analysis. PhD thesis. University of Oxford, 2012.
Kang Y. Visualising forecasting algorithm performance using time series instance spaces / Y. Kang, RJ. Hyndman, K. Smith-Miles // International Journal of Forecasting 33(2), 2017, р. 345–358.
Jin R. Frequent pattern mining in data streams / R. Jin, G. Agrawal // Data Streams. Springer, 2007, pp. 61–84.
Rapach DE. Structural breaks and GARCH models of exchange rate volatility / DE. Rapach, JK. Strauss // Journal of Applied Econometrics 23(1), 2008, р. 65–90.
Gama J. A survey on concept drift ˙ adaptation / J. Gama, I. Žliobaite, A. Bifet, M. Pechenizkiy, A Bouchachia // ACM Computing Surveys (CSUR) 46(4), 2014, р. 44.
Faria ER. Novelty detection in data streams / ER. Faria, IJ. Gonçalves, AC. de Carvalho, J. Gama // Artificial Intelligence Review 45(2), 2016, р.235–269.
Dries A. Adaptive concept drift detection / A. Dries, U. Rückert // Statistical Analysis and Data Mining 2(5-6), 2009, р. 311–327.
Anderson NH. Two-sample test statistics for measuring discrepancies between two multivariate probability density functions using kernel-based density estimates / NH Anderson, P Hall, DM Titterington // Journal of Multivariate Analysis 50(1), 1994, р.41–54.
Duong T. Closed-form density-based framework for auto-matic detection of cellular morphology changes/ T. Duong, B. Goud, K. Schauer // Proceedings of the National Academy of Sciences 109(22), 2012, р. 8382–8387.
Downloads
Published
How to Cite
Issue
Section
License
The authors who publish in this collection agree with the following terms:
• The authors reserve the right to authorship of their work and give the magazine the right to first publish this work under the terms of license CC BY-NC-ND 4.0 (with the Designation of Authorship - Non-Commercial - Without Derivatives 4.0 International), which allows others to freely distribute the published work with a mandatory reference to the authors of the original work and the first publication of the work in this magazine.
• Authors have the right to make independent extra-exclusive work agreements in the form in which they were published by this magazine (for example, posting work in an electronic repository of an institution or publishing as part of a monograph), provided that the link to the first publication of the work in this journal is maintained. .
• Journal policy allows and encourages the publication of manuscripts on the Internet (for example, in institutions' repositories or on personal websites), both before the publication of this manuscript and during its editorial work, as it contributes to the emergence of productive scientific discussion and positively affects the efficiency and dynamics of the citation of the published work (see The Effect of Open Access).
