Research On Efficient Anomaly Recovery Method In Stream Processing System

Stream computing is a core technology in recent years which acquires much more attention in big data processing.At the same time,the stream computing service is also an important platform capability in PaaS(Platform-as-a-Service)system of cloud computing,of which main calculation feature is that it can continuously process the real-time dynamic data stream generated by various network entities.Unlike the big data batch system such as Mapreduce and Pregel,stream computing has a wide range of key capabilities for big data applications such as public service system,enterprise operating system and customer system with online real-time high performance and mass throughput.However,with the expansion of the application scale,the processing load of stream computing system increases greatly,and the probability of abnormal occurrence increases as well,which seriously affects the computational efficiency and application effect of the system.Therefore,how to efficiently recover the anomaly is a problem that urgently needs to be solved in stream computing system.In this thesis,we have studied the abnormal recovery method and anomaly detection method in the stream computing system.The concrete research work and innovation are as follows:1.Aiming at the problem of low efficiency of abnormal recovery in stream computing system,an efficient anomaly recovery method for scalable stream computation system is proposed in this thesis.Firstly,in order to realize the dynamic scale out of stream computing system,the internal states of the working nodes in the system are divided into the input state,logical state,routing state and output state,and upstream backup.When the relevant nodes in the system need dynamic scale out,it is necessary to migrate the upstream state of the node to the new node smoothly.Secondly,in order to realize the abnormal recovery of stream computing system,when the node in the system is abnormal,combined with the tuple upstream backup algorithm and dynamic scale out,the system only needs to dynamically scale out a new node instead of an abnormal node and replay upstream backup tuples on a new node to achieve the efficient recovery.2.Aiming at the passive problem of abnormal recovery in stream computing system,an anomaly detection method for scale out stream processing system is proposed in this thesis.The working states of the nodes in stream computing system are divided into normal,warning,high risk and anomaly according to the frequency of the heartbeat packet of the working node received by the master node.On the basis of high efficiency anomaly recovery,the nodes which in different working states are pretreated.Different levels of abnormal recovery methods can migrate according to the change of the working status of the node.This method can deal with the anomalies of the system in advance and improve the recovery efficiency.In general,this thesis studies the efficient anomaly recovery method in stream computing system from the aspects of high efficiency and initiative.