| As a memory-based computing framework,Spark not only retains the fault tolerance mechanism of MapReduce computing model,but also improves data exchange from disk to memory,that greatly speed up the data processing performance.Spark is also well suited for multiple iterations in machine learning.Spark is very dependent on memory because of its memory-based nature,and the memory useage directly affects the execution speed of the Spark framework.But Spark only uses LRU algorithms for cache management,and the current optimization work is mainly aimed at the cache strategy of the framework.In some cases,there is still have problem of inefficient cache replacement.This thesis researches the cache strategy of Spark and mainly completes the following work:1.For the problem that LRU algorithm dose not consider RDD useage and the cost of rebuild RDD after cache replacement may increase the cost of reloading RDD,this thesis proposes a lowest cost RDD cache strategy to replace the LRU algorithm.This strategy firstly establishes the weight model including the computing cost and makes cache replacement according to the new weight model,and then chooses the lowest cost scheme based on the model,that further reduces the time to rebuild RDD and improves Spark efficiency.And this thesis compares the strategy proposed with weight replacement strategy and Spark native replacement strategy,experiment results show that the strategy proposed can effectively improve Spark efficiency in the case that have large dataset or iterations.2.For the problem that the low efficiency of RDD recovery mechanism,this thesis proposes a strategy for RDD quick recovery in cache based on recovery point.This strategy sets the start recovery point and key recovery point in each stage of the Spark tasks,and serializes these recovery point data.The data is got directly from the recovery point,when RDD is rebuilt from the lineage.That reduce the length of rebuilding RDD lineage and speed up the process of rebuilding RDD.On this basis,this thesis adds a recovery point cleaning algorithm to ensure that this strategy only uses a small amount of disk space during Spark operation and save disk resources.And this thesis compares the strategy proposed with Spark native data recovery strategy,experiment results show thatthe strategy proposed can speed up the process of RDD rebuilding and improve Spark efficiency.3.The lowest cost RDD cache strategy based on computing cost model and RDD cache fast recovery strategy based on recovery point proposed in this thesis are designed and implemented.The research work of the dissertation shows that this thesis research and optimize on Spark cache strategy and select the solution with the lowest cost base on the weight of RDD computing cost in the process of cache replacement,that can reduce the time of rebuilding RDD and speed up the process of rebuilding RDD by serializing important RDD.That can improve the efficiency of Spark,and it is important for Spark to process big data. |
