| Large data computing framework Spark use the memory space,which greatly improve the speed of task execution.Due to the limitation of the memory space,the memory bottlenecks always lead to low efficiency during the execution of the task,even the failure of the task,which is closely related to the framework itself and RDD cache Strategy.Since the large data computing framework Spark was proposed,LRU(Least Recently Used)has been used as its cache replacement algorithm,but Spark’s cache scheduler can not accurately predict the use of the entire task cache,which sometimes leading to poor results.In order to reduce the task execution time,improve memory utilization,by analyzing Spark’s task structure,some optimization are added,and access to the entire task process data and memory usage,through the analysis of the results of optimizing the existing cache strategy.which is the focus of this paper.This paper analyzes the existing caching mechanism of Spark and compares the effects of different caching methods on the performance of the task.It is proved the existing caching strategy has larger optimization space through practical examples.At the same time,the method of task structure analysis and task structure optimization is proposed.For the task of structural analysis,the key information of the Spark task is extracted by the dynamic analysis method,and the dependency graph of the whole task is analyzed according to the dependency relationship between RDDs,and the usage of data and memory during the operation of the task is analyzed.After optimizing the task information of Spark,the task structure is optimized by adjusting the position of the stage.The optimization of the task structure makes the use of the same RDD more concentrated in the task calculation process,which reduces the probability of replacing the memory and improves the execution of the whole task effectiveness.On the basis of analyzing and optimizing the task structure,a reasonable RDD weighting model is proposed to ensure that the relatively valuable RDD in the replacement process can be cached in memory for increased cache hit rate and memory utilization,so we can reduce the calculation error caused by memory bottleneck,which improve the fault tolerance of the Spark framework.Finally,through the comparison experiment,combined with a variety of load use cases,by running a single task,adjusting the cluster configuration,mixing a variety of tasks,etc.,compare with the default Spark and optimized Spark through the experiments,the experimental results show that the task proposed in this paper Structure optimization strategy and cache replacement strategy can effectively improve the efficiency of task execution. |
