Research On Fault Tolerance And Improvement Strategy For Storage Layer Under In-memory Computing Environment

With the rapid development of Internet,the scale of big data is improved dramatically in recent years.In order to cope with the challenges brought by the big data,the in-memory computing with higher execution efficiency has become the focus of concern in both academia and industry.Providing high reliability and efficient storage layer are important for in-memory computing.Through studying fault tolerant computing and optimization of storage layer,we aimed to improve the reliability and memory utilization of the storage layer,and improve the overall execution efficiency of the in-memory computing.For the key problems in storage layer under in-memory computing,the main research work was carried out from the following aspects:(1)The summary of in-memory computingFirst,the concept,classification and typical application of the in-memory computing were introduced,and the concept.Second,the research status of fault tolerance and memory optimization for storage layer under in-memory computing was summarized.Then the concept,architecture,application and problem to be optimized under in-memory computing framework Spark were expound.(2)The research on the automatic checkpoint strategy based on RDD criticality under in-memory computingIn the aspect of the fault tolerance mechanism,the data is recomputed by using lineage in Spark to recovery.However,when the lineage is too long or the operation is wide dependency,the overhead of data recovery will increase significantly due to the large computational overhead.Therefore,it is necessary to set up checkpoints for the tasks in Spark to recovery fast.However,the existing checkpoint mechanisms in Spark require the user or programmer to select checkpoints based on experience,which has a certain degree of risk and randomness.It is not well suited to the task,and may increase the cost of recovery.In view of the above problems,first the job execution mechanism of Spark was analyzed,the definition of the task execution model,the checkpoint model,the recovery model and RDD criticality were established.Based on these definitions,the automatic checkpoint algorithm based on RDD criticality was proposed to reduce recovery overhead,and the cleaning algorithm was presented to clean up unneed checkpoints.Experimental results show that our algorithm can improve the rationality of thecheckpoint strategy,and optimize the task recovery efficiency of the in-memory computing framework.(3)The failure recovery strategy of data under in-memory computing environmentIf exists efficient failure recovery strategy in Spark,it can not only reduce the burden of programmers,but also improve the computational efficiency of the cluster in response to unexpected downtime risk and increase system availability.In addition,the existing failure recovery algorithms do not take into account the characteristics of RDD,the performance of cluster nodes,the data locality and other factors,which may lead to low recovery efficiency.Based on the analysis of the task execution mechanism for in-memory computing framework,the definition of the execution efficiency model consisted of the RDD execution time,computation cost and job execution time was established.The logical relationship between these definitions and the efficiency of job recovery was proved.Based on the semantic analysis of the code and the detailed information of the job,the failure recovery strategy was optimized.It mainly included the calculation of the RDD size and the failure recovery weight,and the usage of the failure recovery model to form the new checkpointing algorithm and failure recovery algorithm.The experimental results show that the proposed algorithm can improve the efficiency of job recovery.(4)The memory optimization strategy based on storage memory minimization algorithm and execution memory assignment on demand algorithmOn the premise of no sacrificing the RDD reuse and job execution efficiency,the storage memory should reduce the space to offer more memory for execution memory.And at the same time,the execution memory should allocate based on the memory demand of the tasks,so as to enhance the efficiency of task execution.The definition of the Spark memory model,memory allocation amount,job execution time,inclination degree,execution memory demand,task memory requirement and task fairness were establised.On the basis of the theory,proposed the strategy of storage memory minimization and execution memory allocation on demand.First,storage memory minimization algorithm selected the more valuable RDD and cleaned the RDDs in time.Then,the execution memory allocation algorithm allocated memory depending on the input data size of the task.With a slight modification of the Spark source code,the allocation of the execution is optimized to reduce the amount of cache dismissed and disk spill rate,and improve the efficiency of task execution.(5)The memory optimization based on small partition grouping and merging algorithm under in-memory computing environmentIn order to reduce the memory consumption and task overhead,the number of storage partitions must be reduced,and it is better to use large partition as input.We considered the memory optimization problem from the perspective of small partition,proposed the small partition grouping and mering algorithm.Through a large number of small partition pretreatment,that is,merged large number of small partitions into a larger file,and then used a large partition as input.In order to improve the efficiency of job execution,it is necessary to reduce the space occupied by the storage layer as much as possible.The experimental results show that the algorithm can improve the effective utilization of memory.(6)The memory optimization strategy based on index structure under in-memory computingIn order to solve the problem of low efficiency and high read overhead,the storage optimization strategy is proposed.This strategy does not affect the original system performance,which using entropy encoding based on sorting and storing.The data in hash storage first ranked according to keys,and the established the index and search tree in the idle time of the system.The strategy can not only provide better service for users to read and access in a short time,but also can reduce the memory reclaim cost of the system.