Design And Implementation Of Batch Processing Job Data Monitoring System On Cloud Platform

In recent years,the application of cloud computing and cloud platforms has become more common,and many companies have built their private cloud computing platforms to meet the computing needs of various departments within the company.To ensure the user experience and improve the efficiency of business processing,large Internet companies need to process a large amount of business-related data every day.The business with low real-time requirements but a large amount of business data will be concentrated at night or at a certain point time at the end of the day for regular batch processing.The volume of jobs submitted for batch processing is huge and it is generally difficult to find a describable job law,which makes it difficult for operation and maintenance personnel to accurately predict the time required for each job to be executed at each computing node.Therefore,it is urgent to establish a scientific and effective real-time data monitoring system to detect the execution time of batch jobs at each node and issue deviation warnings in time to realize automated operation and maintenance.This thesis first investigates the technologies and time-series outlier detection methods related to automated operation and maintenance systems,and an automatic classification-anomaly detection scheme of operation and maintenance is processed.Then,it introduces the related technology theory,details the K-shape clustering algorithm and DBN neural network model,analyzes the relationship between batch jobs based on the K-shape algorithm and DBN neural network model,establishes the batch job classification model,verifies the validity of the model through experiments,and finally applies the established model to job time series data outlier detection.A cloud platform batch job data monitoring system is developed based on the above research results.The main research contents of this thesis are as follows.(1)The overall framework of automatic classification-anomaly detection is proposed.By analyzing the shortcomings in the current operation and maintenance monitoring system,the overall scheme of classification followed by outlier detection is proposed.By classifying operation types through cluster analysis,specific outlier detection algorithms are designed for different categories of operations.Finally,a model is trained to meet the online real-time classification,and the overall scheme is implemented with the designed outlier detection algorithm.(2)Perform K-shape clustering analysis.Detailed analysis of the job timing data characteristics in the dataset is performed.finally,the batch job timing data is clustered using the K-shape clustering algorithm,and the optimal number of clusters K is determined to be 3 by Elbow’s law.The jobs in the batch job data are classified into 3 categories and category labels are added automatically.(3)Training DBN neural network classification model.According to the business characteristics of batch jobs,the structure of the DBN neural network in this thesis is designed.The data is divided into the training set and validation set,and the improved DBN neural network classification model is trained on the training set.Meanwhile,BPNN feedback neural network,RF random forest,and SVM support vector machine model are selected as the control set.The results show that the established DBN model achieves an accuracy of 95.67% on the validation set,which is much higher than the other three types of models,and the delay of the model is 1.8 μs,which can meet the requirements of fast detection.(4)Design of outlier detection algorithm.According to the clustering results of the K-shape algorithm,the fitted time series data of the three types of batch operations are obtained,which are used as the criteria for outlier detection,and the corresponding outlier detection methods are proposed.In the actual operation and maintenance process,the process of outlier detection is as follows.firstly,time-series data is obtained and pre-processed,then the category to which the data belongs is obtained by DBN neural network model.Finally the classification results are input into the outlier detection model,and different outlier detection algorithms are selected according to different job types.(5)System analysis and design.Based on the DBN neural network classification model and the outlier detection model constructed above,the functional and non-functional requirements of the data monitoring system are analyzed,the specific functional design of the data monitoring system is given,and the batch job data monitoring system is designed and implemented.It is mainly divided into four parts,which are the home page display,operation and maintenance personnel management interface job operation status classification query,and historical alarm outlier query.The proposed method and the designed system are validated on the real operation time series data set.The experimental results show that the proposed batch operation classification method based on the K-shape algorithm and DBN neural network model has a high accuracy rate.The accurate detection of time series data outliers is realized based on the classification results,and the designed system has the feasibility of online detection of outliers.The above research provides a reference for time series data outlier detection method selection and detection system design.