Research And Optimization Of Mapreduce Fault Tolerance Mechanisms

With the development of Internet, the size of cloud computing system and thedataset to be processed are becoming bigger and bigger, which increases the needfor a high-performance distributed system with data mining and computationfunctionality. MapReduce is the computation framework of Hadoop, and also thecore component of Hadoop cloud computing. Its fault tolerance is crucial to theperformance of Hadoop cluster. Although MapReduce has a relatively good faulttolerance mechanism and performance, it may degrade the performance in somespecific jobs, due to the distinct requirements in different jobs and scenarios. In thispaper, we optimize and refine original MapReduce fault tolerance mechanism tomake it adapt to various workloads, and enjoy a better fault tolerance power in caseof node failures.We focus on the optimizations of the node failure and data processing in theMapReduce fault tolerance mechanism, which is summarized as follows.Before the task running, estimate the task execution time by prefetching data toperform, and set the time interval of expiration detection (TASKTRACKER_EXPIRY_INTERVAL) based on the prediction, determine the nodewhen the node failure detection time is not received within the timeout node sendsheartbeat back program;In the task running, Every node is configured with acredibility value, which is updated dynamically with the data fetch error andheartbeat report. When the credibility value reaches below the minimum threshold,the node is determined as failed.In the process of running, intermediate valuesstored in the local, it will lost When a node failure, The intermediate data duringprocessing is back-uped asynchronously into different nodes, thus allowing therecovery of the intermediate data in case of node failure. By means of setting thetime interval of expiration detection before the task running, assessing node failuresin the process and Intermediate value storage backup method, enhanced thesystem’s fault tolerance.