Research On Prediction Method Of Anomaly Job In Cluster Environment

Due to the continuous development of big data and cloud cluster computing technology,users can enjoy the convenience and efficiency brought by cloud cluster technology in more and more usage scenarios.In recent years,data security incidents caused by cloud cluster jobs and their computing processes are frequently occured,so it is vital to notice the safety of the cluster jobs and its' calculation process.Therefore,research on efficient and abnormal job prediction methods in cluster environments has become a hot research topic.This paper considered the job anomaly from the perspective of the job's subtasks.Firstly,the basic theory of the abnormal operation prediction method in the cluster environment was described,and the disadvantages of the abnormal operation prediction method were analyzed.Explaining the cluster job's processing mechanism and the correlation between the job's status and its' subtasks' status,In the different stages of the job's subtasks,the support vector machine(SVM)model's prediction principle in the prediction method technology was described and the shortage of the gated recurrent unit(GRU)neural network to predict the anomaly job was analyzed.Secondly,aiming at the problems of low prediction efficiency,long prediction time and no consideration of job relevance in the existing cluster job prediction method,an online cluster anomaly job prediction method based on improved gated recurrent unit neural network(OCAJP-IGRU)was proposed,This method designed the dynamic feature calculation method of the job's subtask in the online phase according to the subtasks' characteristics during its' runtime;then an improved gated recurrent unit(IGRU)neural network was designed according to the dynamic feature to enhance the efficiency of online real-time prediction of the subtask's final status;and then the abnormal job was retrieved according to the job's status relevance.The experiment selected the GRU method,the abnormal frequency threshold learning algorithm(AFTL),the online sequential extreme learning machine(OS-ELM)and the OCAJP-IGRU method to compare the prediction efficiency and the prediction time.The experimental results show that the prediction performance of the OCAJP-IGRU method has higher prediction efficiency and shorter prediction time among the existing online cluster job prediction method under the cluster environment.Finally,in order to further improved the prediction efficiency of the anomaly job prediction method and shorten the prediction time,a staged cluster anomaly job prediction method(SCAJP)was proposed,the offline prediction was plus based on the OCAJP-IGRU prediction process.In the offline stage,selecting the job's subtask static features and using the SVM to predict the subtask's final status based on the static feature then also retrieving the anomaly job based on the job's status relevance,then only online predicting the normal job during the offline stage.The experiments compare and analyze the prediction performance of SCAJP method,OCAJP-IGRU method,SVM method and OS-ELM.The experimental results show that SCAJP method is better than other prediction methods significantly.