Research And Application Of Large-scale Hydrological Sensor Data Anomaly Detection System Based On Flink

With the rapid development of IOT,sensor and communication technology in China,the data scale in various fields is increasing and data has become an emerging asset in the era of big data.Hydrologic sensors play an important role in the field of hydrology and the leap in anomaly detection technology of large-scale hydrologic sensors has also promoted the development of water conservancy construction and economy.Based on the deficiency of traditional hydrologic anomaly detection,a large-scale hydrologic sensor anomaly detection system based on Flink is designed and implemented in this paper.Combined with ARIMA model and Markov chain,the sensitivity and specificity are improved and the computational efficiency is greatly improved by Flink.The use of resource monitoring and benchmarking enables decision makers to select the most appropriate software technology for optimization in a resource context.The main works of this thesis are as follows:(1)Based on the traditional ARIMA model,the ARIMA model can process the flow data by sliding the window.At the same time,the anomaly checking mechanism is introduced.The Markov chain is improved and the one-step transfer matrix is calculated to evaluate the outliers,so that the specificity and sensitivity of anomaly detection are significantly improved.In this paper,the combined model is established by this method,and the feasibility of this method is verified by collecting actual data of Chuhe river and comparative analysis.The results show that the computing time of two nodes is longer than that of one node when calculating millions of data,but the computing time of two nodes is shorter than that of one node when calculating tens of millions of data,with a maximum reduction of 17.43%.The sensitivity of anomaly detection increased from 5.75% to 92.98%.In terms of delay,the average delay of different nodes is roughly the same,all within 20 ms.(2)The time series database and Nosql benchmark are studied.The overall design of the entire Benchmark platform is implemented with a universal kafka-based data interface for synchronously sending messages to different Nosql.The data storage is selected and tested,and the test results most suitable for hydrological data are given.(3)The software composition of the large-scale sensor anomaly detection system is designed and the design scheme of the message middleware in the data acquisition system is established.The communication link between Flink and Kafka is established,and Flink is used for flow processing to realize relevant anomaly detection functions,and the key implementations are demonstrated and explained.Finally,the design of persistence and the running mode of the whole system are described.(4)Regulatory platform is designed and implemented,the connection of regulatory platform and mass sensor is completed,anomaly detection system based on Chuhe river sensor test the basic function of supervision,the results show that regulatory platform can real-time display sensor detecting system data and Chuhe river pictorial diagram analysis,meanwhile it also provides remote for regulators and users of data access and management.