Research And Implementation Of Fault Tolerance And Dynamic Load Balancing Mechanism In BC-BSP System

Graph is the most common type of abstract data structure in computer science, especially suitable for the real world to express various complex relationships. In the field of graph data mining, there are a lot of algorithms require many iterations to get the final result. Aiming at this requirement, large-scale graph processing systems based on BSP model arises at the historic moment. Existing large-scale data processing platform based on BSP model, Pregel, Hama and Giraph although can well deal with massive data, while the platform in a cluster environment is likely to result in system failure due to the unstable factors, and their fault tolerance mechanism also is not the same, it is difficult to come up with a unified solutions. In addition, the system, adoptting Master/Slave architecture, will not avoid to be Master SPOF(Single Point of Failure), and these systems are rarely able to propose effective solutions. Moreover, these systems may also occur to the problem of "bucket effect" due to node hardware problem or node load imbalance. Therefore, implementing the fault tolerance mechanism comprehensively and dynamic load balancing mechanism efficiently is the key and difficult research.According to the above problem, this thesis designed and implemented fault tolerance and dynamic load balance mechanism based on BC-BSP system. The main contributions of this thesis are as follows. Firstly, for BC-BSP system fault tolerance mechanism, this thesis presents a model of the exception processing. The exception processing model has designed perfect scheme for all possible exceptions, and we’ll code these exceptions standardfully. Then, we can process all kinds of exceptions efficiently with the corresponding exception process mechanism. Secondly, this thesis also provides a HA mechanism for BSPController to solve the problem of BSPController SPOF. The HA mechanism adopts hot standby strategy, and BSPController store the backup information on HDFS to keep two node’s information consistenty. Through the Zookeeper service, BSPController HA mechanism effectively implemented the BSPController node failure detection and the active-standby automaticly switching. Thirdly, we designed and implemented the dynamic load balancing mechanism. By selecting the relevant decision informations of all staffs, we can select the short board staff correctly with the scheme of short board staff selection designed in this thesis. Then, we can implement the short board staff migration according to the corresponding migration strategy. By this mechanism, we can realize the migration of short board tasks effectively and change the node load, so as to improve the running efficiency of job.Through the actual deployment and testing, fault tolerance mechanism and dynamic load balancing mechanism based on the BC-BSP system have achieved the desired effect. By BSPController HA mechanism, we can detect BSPController fault and switch automatically in time, and recovery the status of jobs quickly. At the same time, the dynamic load balancing mechanism is also improved the job’s running efficiency, solved the problem of "bucket effect" due to node hardware problem or node load imbalance.