Research On Job Scheduling Method In Storm

With the rapid increase of Internet business data scale,the way of people process data has undergone enormous changes,in order to satisfy people's real-time data processing capacity,streaming computing came into being.Storm is an open source platform for real-time data processing,which can handle streaming data quickly and reliably to meet people's increasingly urgent needs.And the Storm platform scheduler is one of the core technologies of the Storm platform,it has a direct impact on the performance and resource utilization of the Storm cluster.Therefore,the study and improvement of Storm's scheduler has a great significance to the development of Storm platform.The main jobs of this thesis are as follows:First of all,this thesie introduces the related knowledge and the research status at home and abroad of streaming computing and Storm platform,investigatives the architecture of distributed open source streaming platform,and the core technologies associated with topology processing.Secondly,the depth of the schedulers(Default Scheduler,EvenScheduler,IsolationScheduler)provided by Storm platform are analyzed respectively.Topologies are submited to the storm cluster,this thesie observes the performance of three different scheduler when they assigned tasks,and summes up these various schedulers' s characteristics,application scenes and the existence of the problems.And it sets the evaluation index of Storm task scheduling performance,and analyzes the problems of task allocation of Storm default scheduler by experiments.Thirdly,the default scheduler of the Storm platform not only disregards the inter-node and inter-process network traffic,but also disregards the structure of the Topology,the actual resource demand of the task and the resource state of the Storm cluster.In order to solve these problems of the default scheduler,this thesis proposes a resource-aware scheduler based on ant colony algorithm.In our scheduling algorithm,the dynamic changes of the node resource can be expressed as the pheromones of ant movement required,each Foraging ants with its own resource requirements label,and the task scheduling process is similar to ant foraging process,which can improve and optimize the task scheduling of Storm.Finally,the validity of the resource-aware scheduler based on ant colony algorithm is verified by experiments.The experimental results show that our algorithm has strong learning ability,which can find the most suitable node for the current task,achieve the reasonable allocation of resources.And compared with the default schedule of Storm,RSBA algorithm can not only improve the efficiency of task scheduling,effectively reduce the average processing time and improve the throughput of the Storm cluster,but also conducive to the cluster load balancing and optimization performance of Storm cluster.