Design And Implementation Of The Platform For Evaluating Fault Tolerance In Hadoop

With the rapid growth of data in information systems, traditional computingand storage mode is unable to meet the growing demand for data processing andstorage. The cloud computing technology which is based on the early distributedprocessing, parallel processing and grid computing technology becomes a popularsolution of massive data. However, the difficulties and challenges for thiscentralized data solutions also appear with the popularization and promotion ofcloud computing platforms. The reliability of cloud computing platforms is one ofthe crucial aspects. As an important indicator of the reliability of a system,fault-tolerant ability can representatively reflect the level of reliability. Therefore, Itis significant to evaluate the fault-tolerant ability in a cloud computing platform.Due to the increasing complexity of hardware and the increasing scale ofsoftware, evaluating the fault-tolerant ability of a cloud computing platformbecomes a daunting task. Current research results in testing cloud computingplatforms involves less in the evaluation of fault-tolerant ability and contains onlysome simple test methods. It is necessary to make a further research.Fault injection is one of the most effective methods in fault toleranceevaluation. This paper takes the open source project–Hadoop as our research target,studys the fault tolerance mechanisms in the core components of Hadoop. Based onthe failures occurred in real-world applications, it put forwards a multi-level faulttolerance test framework. From software robustness, MapReduce fault injection,network and HDFS fault injection, it simulates a variety of hardware and softwareanomalies in Hadoop deployment structures and operation mechanisms.Based on the fault injection framework, design a Hadoop fault-tolerant abilityevaluation platform, implemet a number of fault tolerance test tools, simulate humanconfiguration errors, node failures, network failures, hard disk failures etc. Then,testers collect the feedback of the cloud computing platforms, analyse the resultsand evaluate the fault-tolerant ability of the systems under test.In order to verify the feasibility of methods in evaluting fault-tolerant ability ofHadoop, it builds a small-scale cloud computing platform with Hadoop. Softwarerobustness test finds and locates the problems in the realization of Hadoop. Theexperiments also show the effectiveness of fault injection tools. With the results ofHadoop benchmarsk before and after fault injection, evaluate the fault-tolerantability of Hadoop.