Research And Improvement On Multi-resource Scheduling Algorithm In Mesos

With the urgent need for rapid development of the Internet and big data processing, more and more domestic and foreign IT companies developed a variety of computing framework based on data-intensive applications. Many companies generally want to deploy these computing frameworks into one clustered environment, and unify resource management and scheduling to let them share data and resources within the cluster.Apache Mesos is a cluster resource management system. It provides two levels of scalable resource scheduling mechanism which makes it easy for the Mesos to be extended. According to the programming interfaces provided by Mesos,it is easy to migrate a variety of applications written in different languages onto Mesost, which can take advantage of the high resource utilization brought by Mesos.DRF is a default resource allocation algorithm provided by Mesos, which can equitably distribute resources in a clustered environment with many types of resources. But DRF algorithm does not consider the inequitable distribution issues due to differences in computer performance. Based on research DRF algorithm which aims at equitably distributing resources, we designed and implemented the meDRF allocation algorithm integrated with the machine performance assessment impact factor. Take the performance score of computing node as the dominant factor of DRF, making computing tasks have equal access to quality computing resources and inferior computing resources. Details are as follows:First, background and significance of Mesos are presented, and a detailed analysis of the system architecture,resource allocation, resource isolation, fault tolerance mechanisms and software ecosystem of Mesos.Second, background of the DRF equitable resource allocation algorithm are detailed, and two important concepts of DRF algorithm: leading resources and dominant share are introduced. The core idea and the realization steps of the DRF algorithm are mainly analyzed. And the implementation process DRF algorithm are analyzed by source code analysis.Last, the inequitable distribution issues caused by ignoring the differences of computer performance will eventually violate the no jealousy feature of DRF algorithm in heterogeneous cluster environment is analyzed. On the basis of DRF algorithm, meDRF allocation algorithm with machine performance assessment impact factor was designed and implemented to increase machine performance. Experimental results show that: meDRF allocation algorithm can better reflect the fairness of the allocation resources and has better stability, which can effectively improve the utilization of computing resources.