Cache Optimization In Spark

Using memory to cache intermediate results is being greatly used in recently memory-based distributed computing engine such as Spark and Tez.With the support of memory cache,there is a great improvement in processing speed of memory-based distributed computing engine.Therefore,the performance of cache has a huge influence on the performance of distributed memory computing engine.In some cases,cache has become a bottleneck for performance improvement.However,cache management policy utilized in prevalent systems are quiet simple and common,notably the Least Recently Used(LRU)spolicy.Although LRU has many good properties,such as low cost overhead,easy to management,and follows the concept of locality of data access,the advantages of LRU set roots on the condition that data access pattern in the future is hardly be identified.If data access pattern can be achieved,making decision based on the future access pattern will gain higher benefits than LRU.The directed acyclic graph(DAG)in Spark can provide access pattern in the future,which can be used in optimizing Spark cache.Recent researches such as MRD and LRC,have made important improvements according to DAG.Based on previous work,we divide the application into online and offline,and then proposes the cache policy for online application and offline application.In the offline scene,we combine the cost evaluation with MRD policy to reduce the useless replacement operations.In the online scene,we take advantage of similarity of different Job's DAG to correct the priority rank in LRC policy.Finally,our experiment with a Spark implementation shows that the method proposed in this paper does have effect.In the best case,compared to LRU,our method speeds up applications by 50%.