Based On Improved Hadoop Yarn Scheduler Design And Implementation Of Large Data Supporting Platform

With the development of cloud computing technology,large data technology also will flourish,the large data technology research has also attracted the attention of academia.At present,there are many large data processing platforms,most of which are based on open source Hadoop distributed platform.The standard Hadoop platform includes many open source components,such as distributed file system HDFS,resource scheduling module Yarn,computing framework MapReduce and so on.With the data more and more diverse styles,as well as the user's data accuracy,processing efficiency,etc.have a higher demand,the existing large data platform has been unable to fully meet the needs of users.In this article,the resource allocation and scheduling mechanism of Hadoop Yarn is studied in detail,and the scheduling process and the scheduling of the dispatching are discussed separately from the resource scheduler.In view of the unreasonable allocation of resources and the lack of queue scheduling,The following programs:1)An adaptive scheduler based on particle swarm and ant colony algorithm is proposed.The scheduler initializes the state and advancing direction of the particle by acquiring the existing task demand and the node resource usage,and introduces the pheromone to be modified.2)Optimize the task assignment of the scheduler.Only the scheduling queue can be intelligent selected by load judgment to improve the scheduling performance.We have improved the traditional Hadoop platform,make it has a variety of data access capabilities,while supporting fixed format text data,relational database and streaming data three types of data.While adding an improved scheduler,making the allocation of resources more efficient and reasonable,while balancing the load of each node.