Research On Adaptive Scheduling Method For Big Data Applications In Hybrid Clouds

The data size of the big data applications increases explosively.The amount of data about financial,weather forecasting and data mining and other areas is up to PB level.Enterprises usually adopt the hybrid clouds to deal with the big data applications.The jobs of big data processed,the tasks of jobs should be executed in the private cloud to make sure of the utilization ratio of the private cloud.The public cloud which could meet the constraints and be with lowest spend should be applied when the resources of private cloud are insufficient in order to achieve the flexible expansion of resources.Therefore,the research on adaptive online scheduling method for various big data applications in heterogeneous hybrid clouds is of great theoretical value and practical significance.According to the above mentioned application scenarios and problems this thesis mainly includes the following three aspects.(1)In order to solve the inappropriate resource allocation of the varied jobs which arrive successively in hybrid clouds,a job scheduling algorithm based on queuing theory is proposed.This paper analyzes the job load type,and jobs are classified according to the Logistic regression method.Taking into account the heterogeneous resources,the resource utility ratio is proposed to classify the nodes of private cloud.According to the job classification and resource classification,the queuing model is established,and the adaptive genetic algorithm is used to solve the job queue's arrival rate which the resources would be allocated based on.The algorithm can reduce the response time of the jobs and increase job throughput in private cloud.(2)The existing time prediction algorithm can not meet the requirements of the tasks of job scheduling in hybrid clouds.Analyzing the factor of the execution time of the tasks,BP neural network is applied to predict the execution time of tasks.Then,a task scheduling algorithm which makes use of improved Max-Min strategy of task scheduling in a private cloud environment based on BP neural network is proposed.When private cloud resources can not meet the needs of the user's QoS,the public cloud which is with the minimum cost and meets the deadline would be applied to achieve the optimal scheduling,improve resource utilization,reduce the response time and save the cost of public cloud.(3)The performance of the proposed algorithm is compared with the existing scheduling algorithm.In the experiment of the adaptive scheduling of jobs based on queuing theory,firstly,the job classification and resource classification and the performance advantage of the improved adaptive genetic algorithm are verified.Then,compared with FIFO,Fair and COSHH,When the system is stable and the number of jobs is 100,the experimental results show that the average job response of proposed algorithm is lower than FIFO,Fair and COSHH up to 76%,56% and 45%,and the throughput of that is more effective than those up to 80%,65% and 31% respectively.In the experiment of the optimal scheduling for tasks based on BP neural network execution time prediction,firstly,to verify the accuracy of the task execution time prediction,the average absolute percentage error(MAPE)of the prediction algorithm is 12.88%.Then,the task scheduling algorithm based on task execution time is compared with FIFO and AsQ.Experimental results show that the performance of the algorithm in terms of task response time is improved up to 59% and 35% compared with FIFO and AsQ,respectively.With the same deadline,the QoS satisfaction rate is higher.And,the algorithm saves 64% and 33% of the public cloud cost compared with FIFO and AsQ,respectively.