Resource Allocation Strategy Based On Data Size Prediction For Hadoop

In recent years,big data processing has become a heated topic of discussion and research in various industries.Load balancing plays a very important role in distributed cluster computing,which affects the processing efficiency of data and the completion time of the job.As a widely used big data software framework,Hadoop has aroused the concern of various industry experts.For example,when a simple Hadoop intermediate data allocation and balancing strategy is handling data with certain characteristics,uneven data distribution may occur which affects cluster processing efficiency and make the load imbalance worse in several cases.To improve the problem mentioned above,this work firstly introduces the background,the history,related works,and the key technology——Hadoop Distributed File System(HDFS)and MapReduce of Hadoop.Then the principle,advantage,disadvantages,optimization goals and existing general balancing strategies in Hadoop have been introduced in detail.After the general introduction,we propose a resource allocation prediction strategy based on data size for load balance which can shorten the completion time of tasks in Hadoop.The main idea of the strategy is to predict the size of intermediate data produced by the Map phrase of Hadoop firstly,then to adjust the resource allocation for every node in cluster.This strategy can not only remain the generality of load balance,but also improve the efficiency of processing.Finally,experiments are implemented to verify the strategy.Results show that the resource allocation prediction strategy can balance the completion time of each task,and shorten the completion time of the job,which improves the efficiency of the Hadoop.