Research On Task Scheduling Algorithm Based On Map/Reduce Model

Cloud computing,as a business computing model that provides services on demand through virtualization of computer resource pools,plays an important role in today's era of Internet-wide data explosion.While constantly providing services for technologies in industries such as science,technology,education and healthcare,cloud computing platforms are also constantly changing the development model and direction of future technical services.In the cloud computing platform,many factors affect the performance of the platform and the efficiency of task scheduling is the most direct and most important.The task scheduling strategy is basically developed based on the Map/Reduce framework.Therefore,the efficiency of task scheduling under Map/Reduce becomes the key factor that restricts the performance of the cloud computing platform.Map/ Reduce nodes in the framework of parallel processing of user submitted tasks.However,the complex task with dependency sub-task has low processing efficiency and the traditional task scheduling algorithm does not reasonably consider the node's computing power and the node's ability to deal with the task,which leads to the low efficiency of the overall task scheduling.In order to improve the efficiency of task scheduling under Map/Reduce,we need to deal with the complex tasks proposed by users and take fully into account the computing power of nodes and the ability of nodes to handle different tasks.To improve the efficiency of task scheduling and resource utilization is the goal of research.The main contents of which are the following:First,improve the Map/Reduce framework: you can add a worker node in front of the Map node for handling complex task,and use these Work nodes for complex tasks,and using this method can improve the compatibility of Map/Reduce.By increasing the order of execution of Worker node.which corresponds subtasks having dependencies to handle complex tasks.Second,the sub-task execution order is obtained by converting subtasks of complex tasks into AOV networks.Then,according to the pruning algorithm,you can achieve the scheduling of complex tasks.As a result of the interdependence of complex tasks,Master converts complex tasks into AOV networks and uses the best topology to obtain the execution order of subtasks.The pruning algorithm is used to match the tasks and nodes.According to sub-task execution order,followed by the Worker node for processing.Thirdly,we use the difference matrix algorithm to realize the scheduling assignment of simple tasks and improve the efficiency of task scheduling.The sampling time is used to get the forecasting time of the sub-task and the ratio of the forecasting time to the computing power of the sub-task is used as the measurement matrix for each sub-task to process the "time".According to the difference matrix algorithm,through first obtaining the local optimal solution to the global optimal solution,we can get the best solution to assign the task,so as to improve the overall efficiency and resource utilization of the task scheduling.Finally,the Hadoop platform is used to validate the preprocessing of complex tasks.The main work is to compare the pruning algorithm with Fair Scheduling Algorithm and Genetic Algorithm in terms of task response time and task execution time.Experiments show that the pretreatment of complex tasks can significantly improve the processing efficiency of complex tasks and the overall accuracy of the task.By using Cloud Sim cloud simulation software to verify the difference matrix algorithm,the difference matrix algorithm is compared with Fair Scheduling Algorithm and Genetic Algorithm respectively from two aspects of the overall efficiency of task scheduling and the utilization of cluster resources.Experimental results show that the difference algorithm can significantly improve the overall efficiency of task scheduling and cluster resource utilization.