Process-oriented Perception Of Cloud Resource Scheduling Jobs

Development of cloud computing has brought many changes to society, social life is constantly digitized. How to handle vast amounts of data has become a more and more fashionable topic. Hadoop is a distributed computing software framework, which includes HDFS and MapReduce distributed computing method, make distributed processing huge amounts of data possible. Then job scheduler determines the efficiency of Hadoop clusters and user experience. However, current scheduler does not take into account the heterogeneity of the cluster, how to schedule job efficiently on Hadoop heterogeneous clusters, has become a challenging problem.Under the premise of familiar with the mechanism of Hadoop’s running tasks,make a full analysis of the existing FIFO-Scheduler,Capacity-Scheduler,FairShare-Scheduler and LATE-Scheduler, found that the existing scheduling algorithms do not perceive the performance of the computing node, so that it can not assign different tasks depending on the machine performance in heterogeneous Hadoop cluster. In this paper, we propose a process-aware task scheduling algorithm named IOAware. The algorithm would like evaluate the performance of hardware of computing nodes and speculate property of task during the execution process of task. At the time of the follow-up assignments, the algorithm would assign different tasks based on the performance of TaskTracker and the property of task. so as to achieve shared computing node disk10effect. This can shorten the execution time of the task in parallel, to improve throughput of the cluster.Feautures of IOAware algorithm is reflected in two aspects. One is the demand for disk10of the task determine which type the task would be. There are two types for the task. One is CPU-Bound, the other is IO-Bound. Put different types of tasks together, reducing disk10operation at the same time and the possibility of disk obstruction. The other is improve the rate of task’s input data localization as an important indicator to reduce the data network transmission time, so that reduce the execution time of task.In order to verify the theoretical feasibility of scheduling algorithm IOAware, we proposed schedule model and realized IOAware scheduler. The scheduling module was used in Hadoop cluster for multiple experiments. From he response time of the job, the system throughput and the use of system resources, compare IOAware with FIFO scheduler, Capacity scheduler and Fair scheduler. Through experiments found that the execution time of a single task, the scheduling module is consistent with the existing scheduling module obtained; For tasks with different attributes, scheduling module would assign different tasks on the same TaskTracker, so that reducing the number of disk operations at the same time, reducing the CPU waiting time for the disk and improve CPU utilization, effectively raising the rate of data localization tasks, and enhance the throughput of the system.