Research On Anomaly Monitoring And Recovery Strategies For Stream Processing System

Stream Computing has become an important part of the big data ecosystem.The increasing number of real-time processing scenarios provides a major opportunity for development and application of streaming computing.The main computational feature of stream computing is that it can process dynamic real-time data streams generated by data traceability with milliseconds or even shorter delays.Different from traditional big data batch processing technologies(such as MapReduce and Spring Batch),stream computing technology can enable financial service system and Internet background processing system to obtain the key capacity of real-time big data application with high throughput and low delay.The diversification and complexity of data traceability and the proliferation of various structured and unstructured data bring serious challenges to the load capacity of stream computing systems.Along with the increasing processing load,the possibility of abnormality in the stream computing system is greatly increased,and its credibility and usability are difficult to reach people's requirements.Therefore,it is very necessary to conduct in-depth research on the anomaly monitoring and recovery strategy of the stream computing system,which is also a key requirement for enhancing system availability.This research is based on the field of anomaly monitoring and recovery in the stream computing system,which is relatively lagging behind.This paper starts the research of basic theory and technology,and the specific research work and innovations are as follows:1.Through the analysis of the convection computing system,the key indicator data that can accurately express the system's operating status is studied.The basic indicators of system operation are collected and selected as the basic research data set,and an anomaly detection method based on hidden markov model for stream computing system is designed.First,collect indicator data that reflects the operational status of the stream computing system and perform pre-processing.Secondly,a statistical monitoring model based on hidden Markov model is established.The external characteristics of the system are linked with the internal state through Hidden Markov Model,so that the model can adapt to the streaming computing service with synergy,state preservation and transformation.Finally,based on the monitoring results,the operational status of the convective computing system is simply classified to form a multi-class monitoring mechanism.2.In order to overcome the disadvantage of low fault tolerant recovery efficiency of stream computing system,this thesis designs a multi-level abnormal recovery strategy based on node extension.Firstly,the recovery strategy adopts the method of tuple reduction to reduce the backup tuples,thus reducing the recovery time.Secondly,the basic recovery mode is formed by combining the extensibility of the nodes with the upstream backup strategy of tuples.When an exception occurs on a node in the system that needs to be restored,simply replay the tuple on the newly extended node.Finally,according to the abnormal state of the node,it is divided into different levels and forms multi-level recovery,which can sense the system's abnormal to a certain extent and improve the efficiency of abnormal recovery.3.Based on the above two researches,this thesis builds the corresponding experimental platform based on the existing experimental conditions.Based on this,the fault or abnormal injection method is used to analyze the effectiveness of the relevant anomaly monitoring method and recovery strategy.In summary,this research starts from the two key points of the anomaly monitoring and recovery strategy of the stream computing system,and studies the credibility and reliability of the stream computing system.