Peer-Comparison Based Fault Diagnosis System For Hadoop

In the age of big data,more and more data are produced in corporation,government and research institute.In order to handle such massive amount of data,MapReduce as a tool to process data sets in a parallel way on cluster was developed by Google.MapReduce can effectively divide huge datasets into small parts evenly and distribute them into every single node on cluster for processing.Hadoop MapReduce,the open source implement of MapReduce,has been used in many organizations including Facebook,Yahoo,etc.With the increase of data,more and more machines are deployed on clusters.It becomes difficult to identify and diagnose faulty nodes suffering problems which would lead to continuing running but with degraded performance.How to diagnose such kind of fault has become interesting problem.We present a diagnostic system which is based on analysis of Hadoop logs and OS-level performance metrics that can be applied to diagnose such kind of performance problems in Hadoop cluster.This technique is based on hypothesis that faulty-free nodes in Hadoop cluster behave similarly.Automatic diagnosis module can indict those nodes with asymmetric behaviors as faulty nodes and then identify root cause of these problems further and visualization module works as a supplemental tool for the system.It represents visualization of Hadoop behavior from three different aspect,including tasks,data and performance metrics.Users can conclude the exsist of performances problems from these diagrams.In addition,the experiments have been conducted in order to validate the similar behaviors of nodes in fault-free Hadoop cluster and the effectiveness of our approach.