Design And Implementation Of Distributed Stream Computing Framework Fault Tolerance

Streaming computing is the study about real-time data stream processing, and its idea is that value of the data is reduced with the lapse of time, so as long as the data appearing, it must be processed as soon as possible. The best tactics is that data be processed immediately after appearing, rather than saving it. In various solutions, distributed stream computing framework has been widely welcomed by the industry because of its good scalability, ease of use and flexibility. This article’s purpose is to design and realize the most critical function fault tolerance of distributed stream computing framework.First, based on the distributed stream computing framework research background and current research trends, elaborating the significance of fault-tolerance for distributed stream computing framework, and proposing the idea of designing and realization of fault tolerance, and introducting the key technologies needed to realize the fault tolerance function.Next, based on the open source distributed stream computing framework S4(Simple Scalable Stream System), and elaborating functional requirements analysis for fault-tolerant of distributed stream computing framework, which mainly consists of three parts:processing node fault-tolerant features, the fault-tolerant features between processing nodes communication and the calculating fault tolerance for processing node. The fault tolerant for processing node is to achieve the reliability of the node itself, processing node networks tolerant feature is to achieve the reliability of the network communication between processing nodes, calculating fault tolerant is to achieve the reliability of the result of application processing.Finally, detailed design and implementation of the fault tolerance of distributed stream computing framework is carried out in the following. The fault tolerance of the processing nodes through applying replication technology which regular save processing node status information to a third-party database MongoDB, and when processing nodes need to be recovered, the node’s recently saved information is obtained from MongoDB and initializing the newly created node; fault tolerance of network communication between processing nodes through using open source network communication library ZeroMQ, which can handle message error and transient failure of network connection, the fault tolerance of calculation for processing nodes through employing multi-version software fault-tolerant technology which can shield system’s various failures to a certain extend. At the end, the paper focus on main fault tolerance module’s testing and analysis, experiments shows that fault tolerance has achieved better reliability and robustness for the system, at the same time, a brief summary of the paper’s improved aspects for the future is elaborated.