Research On Anomaly Detection Of DOA Data Based On Deep Learning

With the advent of the era of big data,the importance of data for the development of the country and society has become increasingly apparent.In order to make data play its greater value and reduce people's difficulty in obtaining,Professor Miao Fang proposed a data-oriented architecture(DOA,Data-Oriented Architecture),which adopts the idea of "data-oriented and data-centric".DOA is mainly divided into data registration center(DRC,Data Register Center),data anomaly center(DEC,Data Exception Control Center),data authority center(DAC,Data Authority Center)and data application units(DAUs,Data Application Units).The data anomaly center is an important module.Through the collection and detection of system and application logs,the detection and alarm of anomaly logs are realized.In this paper,by studying the deep autoencoder log anomaly detection algorithm,an automated log anomaly detection model without manual labeling is proposed,and combined with the ELK(Elasticsearch,Logstash,Kibana)framework widely used in enterprises,a set of log collection,log cache,log processing,log storage and log is built for DEC Show as a log platform.The main research contents of this paper are as follows:(1)Research log analysis and feature extraction algorithmsResearch log parsing algorithms,specifically including AEL(Abstracting execution logs),IPLo M(Iterative Partitioning Log Mining),and Mo LFI(Multiobjective Log message Format Identification),to solve the problem of converting unstructured text text log data into structured data data,and convert text log data into log templates and variables.Research log feature extraction algorithms,including feature extraction based on fixed windows,feature extraction based on sliding windows,and feature extraction based on conversation windows,to solve the problem of converting log templates and variable data into a digital matrix that can be trained by the model.(2)Research on autoencoder network model for log anomaly detectionStudy autoencoder network models,including common autoencoder network models and variational autoencoder network models.In order to solve the problem that the commonly used machine learning model has no significant effect on multiple log anomaly detection,an automatic log anomaly detection model without manual labeling is realized.(3)Research log platform construction technologyResearch the construction technology of log platform,including ELK log framework,Kafka message middleware,Flink streaming and Flask webserver framework.Implement the DEC log platform that integrates log collection,transmission,processing,detection and warning,storage and display.The main research results and innovations of this paper are as follows:(1)The log anomaly detection model of the joint autoencoder network is proposedFor the problem that log anomaly detection needs to mark samples,this paper proposes the log anomaly detection model(UAE,union autoencoder network)of the joint autoencoder network.The autoencoder network model is a representative deep learning model,which has significant advantages in feature extraction and generalization.This article first analyzes the original log and converts the unstructured log data into a data matrix.Then,the dimensionality reduction is performed through the encoder layer of the autoencoder network model.The dimensionality-reduced data is detected by the IForest(Isolation Forest)model,and the data detected by IForest as a positive example is then trained by the complete autoencoder network model to achieve the purpose of training the autoencoder anomaly detection model without marking the anomaly log.(2)Proposed an anomaly detection model based on pre-labeled depth variational autoencoderThe deep variational autoencoder anomaly detection model makes anomaly judgment through the reconstruction probability of the model.However,due to the resampling of the encoder layer,it is impossible to use the similar method of the joint autoencoder network to detect the dimensionality reduction data.Therefore,this paper proposes a deep variational autoencoder anomaly detection model(PVAE,pre-labeled depth variational autoencoder network)based on pre-labeling.It first uses the CLOF(Clustering-based and LOF Outlier Detection Method)model to pre-label the original data,and passes the pre-labeled positive data to the deep variational autoencoder anomaly model training.And use the integrated learning method to model fusion of UAE and PVAE models,integrate the recognition results of the two models,as long as one of the models is predicted to be abnormal,it is judged as an abnormal log.(3)DEC log platform was builtUse the ELK framework to build a log collection,transmission,processing,storage,and display platform.And through the Flink streaming processing framework,the UAE model and the PVAE model are used to detect log anomalies at the log processing layer.If an abnormal log is detected,an alarm email will be sent to the management personnel.