Researches Of MapReduce Parallel Programming Model And Scheduling Algorithm For Heterogeneous Multi-core Systems

With the increase of the difficulty of parallel programming in heterogeneousmulti-core environment, the parallel programming model can handle and generatevery large scale data sets is needed urgently. Such model can reduce the difficulty ofparallel programming, so that the development rate of heterogeneous multi-coresystems can be increased.MapReduceis a parallel programming model which is emerging in recent years.The model is mainly used to achieve the computing tasks of the sub-division of tasks,computing resource scheduling, computational structural reduction. MapReduce is asimple, elegant solution for heterogeneous parallel system of data processing.However, those scheduling algorithms of the traditional MapReduce have somelimitation such as its response time is too long and system’s throughput drops sharplywhich affects the performance of the whole system. Based on the deep study of theMapReduce parallel programming model, the paper proposed a novel heterogeneousmulti-coreMapReduce scheduling algorithm which is suitable for the Hadoop platform.following are the main work:(1)Based on the scheduling problem of the MapRedcue model, the papersummarized and analysed the three factors: locality, synchronization and fairnessconstraints, which make the main effect for the algorithmic efficiency. Furthermore,to study the execution of handling the synchronization cost in MapRedcue, the paperdiscussed two methods which are asynchronous processing and speculative execution.Then, locality improvement, delay scheduling in Hadoop and quincy scheduler inDryad are discussed.(2)The paper proposed an novel improved heterogeneous multi-coreMapReducescheduling algorithm,based on the characteristics of heterogeneous multi-coreenvironment, considering the weakness of the typical MapRedcue schedulingalgorithm—LATE algorithm. The novel method adds the ability to obtain theregulatory learning of Machine Learning and randomly gathers some task as test taskto get the information on the processing nodes. Then it gets the time percent of eachstage and adjust the the way applications run, then lunches the backup task to reducethe response time.In order to prove the effectiveness of the proposed algorithm, Based on theHadoop platform, experiments about the algorithm are performed. The result shows that the algorithm is better than LATE algorithm and the original algorithm of theHadoop platform. All in all, the proposed algorithm is good for increasing theperformance of the whole system, and it also has some promoting significance forheterogeneous multi-core parallel computation.