Design And Implementation Of Anomaly Detection System For Mesos Cloud Platform

As cloud computing becomes more and more mature,container-based lightweight PaaS platforms are favored by major operators due to theirs large-scale rapid deployment advantages.With the increasing complexity of the cloud platform and the frequent occurrence of “cloud shock” events,the security and reliability of the cloud platform are threatened.This paper uses the anomaly detection system to find out the corresponding anomaly issues to provide a basis for the administrator to resolve the anomaly,and then improve the security and reliability of the cloud platform.How to reduce the data redundancy dimension,improve the mark accuracy rate,reduce the waste of unmarked data,improve the accuracy of anomaly detection algorithm and improve the abnormal position of the administrator is an urgent problem to be solved in cloud platform anomaly detection.This paper designs and implements a cloud platform anomaly detection model.The model can locate anomalies at the resource level,detect the anomaly detection model based on various types of resource data of the container,and then collect statistics from physical machines,tasks,and resources.The analysis provides a basis for the cloud platform administrator to quickly locate anomalies.Firstly,abnormalities detected through the cloud platform container include: no data interaction for a long period of time in the container,failure to complete the corresponding task within a specified time,dead-time feedback of the container,etc.,and then sum up the corresponding abnormality discovery rules.Secondly,for the problem of dimensional redundancy and low accuracy of data mark for a large number of container data,data preprocessing and data labeling methods are proposed.Data preprocessing uses information gain for container data to extract important data dimensions.The data tag positions the container exception tag to the resource level.Thirdly,aiming at the problem of low accuracy of anomaly detection model and waste of unlabeled data,a semi-supervised random forest anomaly detection model is proposed.The training data includes most of the marked data and a few unlabeled data,and iteratively performs random forest and data editing.In this paper,K-means clustering and K-nearest neighbor ensemble learning model based on attribute weighting are proposed in data clips to improve the accuracy of data editing.By constructing the anomaly detection model,we design and implement an abnormality determination and root cause analysis module.Then,through statistical analysis of the data,a statistical analysis module for tasks and resources are designed and implemented to provide a basis for administrators to solve abnormal problems.Finally,the anomaly detection model is used to implement the system anomaly detection function;design and implementation of abnormality determination and root cause analysis,tasks and resource statistical analysis modules;based on data preprocessing and data tag rational design of the database table.Run the black box and white box test methods to test the system's key functions such as abnormal judgment and root cause analysis,task and resource statistical analysis,as well as system performance,test and verify system usability.In summary,this paper designs and implements an anomaly detection system for Mesos cloud platform.The accuracy of the proposed anomaly detection model is 86.2%,which is 4%-5% higher than other anomaly detection systems based on PCA and information entropy,and improves the administrator's abnormal positioning speed and achieve system design goals.