Research On Adaptive Checkpoint Mechanism For Large-scale Streaming Data Processing

With the development of cloud computing,blog,social networking and other emerging network services,streaming data processing has become a hot technology nowadays.The research of static large data batch processing is relatively mature.However,the research of online processing of large scale streaming data is still in the initial stage,streaming data becomes the important data.Different from static data,streaming data has the characteristics of continuous updating,large data scale,timeliness,dynamic change of workflow and burstiness.Based on the early stage of stream processing systems,many distributed computing systems can be real-time or near real-time to process large data stream(S4,Storm and Spark Streaming)with parallel processing ability.Importantly,the system has strong scalability.Different from the computer cluster and static batch processing system,the most remarkable characteristic of the distributed stream processing system is real-time processing ability.At present,the distributed stream processing system needs to improve the fault tolerance urgently: Firstly,streaming data is dynamic and flows into and out of the system during the process of continuous data,usually relies on memory computing.What's more,these lead to permanent loss of data information in the fault,increasing the difficulty of maintenance.Secondly,the scale of distributed systems is strong and the system failure rate is rising,which lead to the aggravation of reliability problem.Thirdly,the workload changes dynamically and the stream processing job has both data state and computing state.Combining the characteristics of stream computation,checkpoint and upstream backup are more effective fault tolerance methods.However,the existing checkpoint mechanisms for distributed stream processing systems are mostly based on traditional methods.The existing checkpointing scheme is relatively simple,fixed periodic,which is short of comprehensive consideration of factors such as changes in the data burst.If the workload peak occurs,the implementation of a checkpoint will seriously affect real-time processing.Due to the lack of a more systematic theoretical model,it is difficult to predict and determine the data source and traffic changes,there is no relevant technology to quantitatively analyze the impact of the checkpoint on real-time data streams under different workload scenarios.With the bursty stream,the traditional fixed checkpointing method is difficult to provide time guarantee for fault recovery.Therefore,the fixed checkpoint is not fully applicable to the current distributed streaming data processing platform,it is difficult to meet the reasonable balance between delay requirements and fault recovery time.The real-time fault tolerance of distributed streaming data processing system has strict requirements for processing delay and fault tolerance.This paper presents a checkpointing mechanism which can support data stream load change and on-line adjustable period.Firstly,for the bursty stream scenario,a recovery time calculation model is established,which provides fault recovery time guarantee for node failures.Coordinating upstream and downstream nodes for checkpointing,we avoid the possible memory overflow problem.Secondly,provide a periodic dynamic adjustment scheme.According to the real-time change of data stream,the real-time quantitative model of checkpoint is proposed,which quantitatively describes the delay influence of checkpoint on data processing.In addition,a checkpoint period management scheme is designed,and the checkpoint period of each node is analyzed in the classification management.Moreover,an adaptive period adjustment protocol is proposed,which is based on the real-time cost of checkpoint to select the best checkpoint opportunity for nodes dynamically.Finally,the experimental results show that compared with traditional checkpointing methods,this mechanism has obvious advantages in flexibility and real-time,which can meet the requirements of high reliability and real-time fault tolerance of streaming data processing.