Design And Implementation Of Resource Management System Of Big Data Management Experiment Platform

In order to ensure the stability of the big data experiment management platform and prevent students from making large differences in the workload of different physical hosts during the experiment,resulting in problems such as low resource utilization rate and unbalanced load of the experiment platform,we need to manage all types of resources in the platform in an effective way to improve resource utilization.By monitoring and scheduling the various resources of the platform,a resource management system for a big data management experiment platform is designed,which is mainly applied in the teaching environment where student users conduct big data experiments.The system mainly includes a cluster management module,a hierarchical finegrained resource monitoring module,and a system resource scheduling module.The cluster management module is mainly to provide users with convenient cluster management related functions.The hierarchical and fine-grained resource monitoring module monitors the resources of physical hosts,virtual machines,and student users.User monitoring includes the monitoring of student user resources and the monitoring of student user behavior.In system resource scheduling module,resource scheduling strategies has been designed in different scenarios.The resource scheduling is triggered by resource warning,including virtual machine initial placement,virtual machine resource adjustment,and virtual machine migration.A virtual machine initial placement algorithm based on multi-dimensional resource loading is used to select the most suitable host node for the virtual machine.A virtual machine resource adjustment strategy and a dual-threshold virtual machine migration strategy based on weights under resource warning are designed.Under different resource distribution situations,the corresponding scheduling strategy is selected for resource scheduling.The experimental results verify the effectiveness of the initial placement of virtual machines,the adjustment of virtual machine resources,and the scheduling strategy of virtual machine migration during resource scheduling.Through resource scheduling,the resource utilization rate of each physical node is improved,and finally the load balance of each physical node is realized.