Research And Application Of Outlier Detection Algorithm Based On Density & Neural Network

With the fierce development of the networks,massive data will continue to be formed.How to obtain hidden information from the massive data has become an important task of the data mining.Outlier detection technology occupies a large proportion in the field of the data mining.The goal of outlier detection technology is to find data objects that are significantly different from most objects in the data set.Outlier detection technologies have a great many applications in the domains of abnormal weather forecasting,astronomical observation,and network intrusion detection.There are some technical challenges in the research of outlier detection technology,such as detecting outliers at the edge of dense clusters or extracting feature relationships in the detection data set.In this dissertation,a new model based on the idea of density-based outlier detection algorithm,or optimizes the defects of CNN network model in the field of outlier detection,so as to improve the detection accuracy of outliers in the model.The detection accuracy is lower and the false alarm rate is higher in the area with the characteristics of dense clusters or normal objects outliers in areas with dense cluster characteristics or normal objects in high-dimensional or dense cluster data sets by analyzing existing outlier detection algorithms.An outlier detection algorithm based on an equal-area selection strategy: EASOD(Equal Area Selection Strategy Outlier Detection)is proposed.Firstly,the EASOD algorithm selects the radius measure by using equal area division strategy in the pending datasets;Then finds the neighborhood of the objects according to the radius measure;Secondly,calculate the outlier score of the data objects;Finally,the top-n objects with highest outlier value are output as outliers.Comparative experiments between EASOD and LOF,LDOF,IForest about detection rate and error alarm rate are carried out.The proposed algorithm has a higher detection rate when the error positive rate is low.The results demonstrate that EASOD achieves better performance in synthetic datasets and real datasets.An outlier detection algorithm based on multi-scale convolution feature autoencoder: SCAOD(Multi-Scale Convolution Feature Autoencoder Outlier Detection)is proposed to solve the CNN model that requires training data of balance in outlier detection.By combining shearlet transform with CNN,the SCAOD algorithm can enhance the feature extraction ability of CNN.The autoencoder is used as the anomaly detection module,and the anomaly fraction is judged on the full connection layer after CNN extraction of multi-scale features under shearlet transform,and the final outlier set is obtained.The SCAOD algorithm is compared with the four comparison algorithms on eight real data sets,and the results of its indicators show the effectiveness of the SCAOD algorithm in detecting outliers.