Reliability Analysis Of Hadoop Cluster System Based On Cox's Proportional Hazards Model

Nowadays,Hadoop clusters have been widely used in the big data era and Internet companies for data processing and analysis.Once the Hadoop cluster fails,it can lead to data loss,bring the inconvenience to work,and even bring significant economic losses.For the country and the individual,the reliability of the Hadoop cluster is always a headache problem.Thus,the research on reliability of Hadoop cluster is very necessary.There are several Data Nodes and NameNodes mainly contained in the entire Hadoop cluster.When the reliability of the nodes is analyzed,software failures are mainly related to the load.Therefore,this paper uses Cox's proportional hazards model to describe the relationship between node failure rates and load(cumulative load and instantaneous load).In the Hadoop cluster operation,node failures will lead to software degradation.So the Hadoop cluster failure process can be seen as a non-homogeneous Markov process(NHMP).In this process,the software load is not fixed,it changes over time.The process can be divided into several non-continuous non-homogeneous Poisson processes(NHPP),which are used to describe the system reliability process before each node fails.On this basis,this paper analyzes the reliability of Hadoop cluster.The cluster reliability model proposed in this paper can be used in cluster design and research.Finally,the author conducted an evaluation experiment to demonstrate the feasibility of the method.