Research On Cluster Resource-Efficiency Oriented Prediction Of Host Load And Job Status

The low availability of cluster resources is an important challenge facing the current cluster.The specific manifestation is that the hosts in the cluster fail to obtain proper resource allocation,and there are a large number of unsuccessful jobs that are repeatedly submitted and run,all of which lead to a lot of waste of cluster resources.In order to solve the above problems,this article focuses on the problem of host load prediction and job status prediction.Researchers have done a lot of work to address the above two issues.This article summarizes the two major shortcomings of previous work.One is that traditional methods cannot make sequence predictions for the host load,and it is easy to produce cumulative errors when predicting actual load;the other is that the data sets of previous research work are limited in scale and quantity.It is easy to overfit on a single cluster,which weakens the universality of the analysis results.Based on multiple public cluster load tracking data sets,this paper extracts and analyzes the host resource load characteristics and key factors affecting the job termination status,and finally proposes a host load prediction method and a job status prediction method.This paper carries out research and analysis from the perspective of theory and experiment respectively,and specifically does the following work:(1)Analyzed the host resource usage of different clusters,express the characteristics of cluster host resource load from different angles,and propose a host resource load prediction method.Describes the exponential segmentation prediction mode of the host average load prediction,predicting the average load value in the future continuous time interval and the actual load change value in the future time period in advance.(2)Analyzed the job termination status distribution and resource usage characteristics of different clusters,designed three features to describe the real-time display of job status,and combined the static and dynamic features of the job to effectively predict the future termination status of the job.