Design And Implementation Of Security Data Stream Anomaly Detection System Based On Stream Processing

China's securities market is constantly developing and prospering,new enterprises are listed in the domestic market,and the volume of transactions is also growing.The expansion of its scale also promotes the development of China's real economy.However,because of the great changes in politics,industry and public opinion,the stock market will fluctuate,and the stock market has the characteristics of linkage and prone to occur,which will react on the real economy and hinder the economic development.Therefore,it is necessary to assess the current risk of China's securities market,real-time supervision of the abnormal behavior of the securities market,the sooner we find the abnormal situation of the securities market,the faster we can deal with it.However,today's securities regulatory authorities are facing business pain points: 1)the large amount of data causes high latency,which can not meet the rich market demand with high-speed changes;2)the threshold of anomaly detection rules depends on the long-term business experience accumulated by business experts,and can not be corrected in time with the changes of the market.With the maturity and stability of distributed systems,the emergence and development of stream computing technology in recent years,and the emergence of various machine learning equivalence algorithms,we consider building a security data stream anomaly detection system based on stream computing,and design real-time data source access,real-time data preprocessing,Real-time Anomaly Detection and real-time anomaly detection.Four modules are introduced to detect anomalies in securities data stream and achieve real-time second alarm,so as to improve supervision efficiency and ensure real-time anomaly detection.Specifically,the following work has been done in this paper:Firstly,this paper analyses the real-time growth securities data generated by the trading system,tries to use the real-time flow computing framework to access the data,and designs the whole system architecture which uses Kafka as the data bus and stream computing as the implementation method.Secondly,based on Flink framework,the access of real-time data sources and the design and development of real-time data preprocessing are implemented.Then,flow computing is used to support real-time anomaly detection based on expert's anomaly rules.Autoregressive moving average model and hierarchical real-time memory cortical learning algorithm(HTM)are used to detect anomaly in securities data stream.Finally,the input source and exceptional results are persisted to the ground inreal time,and the exceptional results are pushed by subscription.In the aspect of testing,from both functional and non-functional aspects,the test cases are given and completed in stages.In addition to the functional assurance in the requirements,the delay and reliability of the system are also tested.This system realizes real-time anomaly detection of real-time securities data stream,provides faster supervision information for the market supervision department,and provides early warning for investors.