Spark Fault-tolerant Optimization Based On Compensation Function

In the era of large data,with the increase of data volume and the exploration of data value,distributed big data computing system has been widely used and studied by enterprises and institutions.With the increasin g number of distributed system nodes,the failure rate also increases,and fault tolerance becomes a key technology that can not be ignored in distributed big data computing system.In the field of large data applications,especially data mining and machin e learning,iterative computing has become a major feature of its algorithm,which through the iterative process to achieve the purpose of solving the optimal solution.Spark,as an emerging universal data processing framework,is based on memory computing and has excellent performance in iterative computing,and has rapidly become the most popular distributed large data computing platform.However,Spark mainly uses Lineage mechanism to achieve data fault tolerance,Lineage records how a data set evolved from other data sets,when a block of data is lost,Spark through the record of the Lineage information back to the loss of data dependencies,recalculate the loss of data,In the iterative calculation of the equal length task scenario,there is a problem that the recalculation recovery time is too long.In this paper,the iterative calculation process and its convergence are analyzed,and the iterative computation has the stability from different state convergence.An optimistic fault-tolerant mechanism based on compensation function is proposed to realize the fault tolerance of the data and this mechanism is used to optimize Spark's fault-tolerant.Which is different from the traditional fault recovery method of recalculating the data.This mechanism compensates the lost data quickly through the defined compensation function when the data loss occurs,rather than recalculating the original data and ensuring the consistency of the whole data set,So that the algorithm can continue to perform,through the subs equent iterative process to correct the data,and converge to the correct results.In the absence of failure,this mechanism uses optimistic fault-tolerant mode,do not add any faulttolerant measures,will not cause additional overhead.The experimental results show that the optimistic fault-tolerant mechanism based on the compensation function is effective to guarantee the reliability of iterative data,and the performance is better than the existing fault-tolerant mechanism.