Research On Anomaly Detection For High-dimensional Data Based On Deep Learning

Anomalies,which are also commonly referred to as biases or outliers,refer to that are different from norm patterns or normal data.Since anomalies can convey valuable information about the data,anomaly detection is considered to be an important link in decision systems.The data obtained in applications is complex and high dimensionality,currently,anomaly detection of high-dimensional data has two challenges.(1)High dimensionality leads to anomalies difficult to be detected.Anomalies show obvious anomalous features in a low-dimensional space but become insidious in a high-dimensional space.The high-dimensional space contains many subspaces,given enough candidate subspaces,at least one feature subspace can be found for each data point,thus increasing the complexity of anomaly detection.(2)Irrelevant attributes in a high-dimensional space can create noise of masking anomalies so that anomalies and noise are difficult to be identified.It is difficult to select a subspace to highlight relevant attributes in an exponential search space,so that identifying irrelevant attributes is the core of dealing with the curse of dimensionality in anomaly detection.Aiming at the above two challenges,by analyzing the mainstream anomaly detection methods,and applying related theories and deep learning methods,four anomaly detection methods for high-dimensional data are proposed,including an anomaly detection method based on a deep hypersphere,a deep anomaly detection method based on the optimal transmission theory,an anomaly detection method based on conformal prediction and an anomaly detection method based on conformal transformation with fuzzy.The main contributions of the dissertation are as follows:(1)Aiming at the problem that anomalies are difficult found caused by high-dimensional dimensionality,an anomaly detection method based on a deep hypersphere is proposed.The proposed method uses deep network structures to capture low-dimensional features from the high-dimensional data,then uses the hypersphere to separate anomalous features from the captured features.To learn compact boundaries separating abnormal and normal features,the angle kernel and the radius kernel are derived.Experimental results show that the proposed method outperforms competing methods in anomaly detection accuracy and the ability to learn compact boundaries.(2)Aiming at the problem of anomaly detection for irrelevant attribute interference in a high-dimensional space,a deep anomaly detection method based on the optimal transmission theory is proposed.To realize the reconstruction of the feature space,using the optimal transmission theory implements itself homeomorphic transformation for an unknown original feature space.Abnormal features,normal features and irrelevant attribute features are separated in the reconstructed feature space.To prevent irrelevant attribute features from being misjudged as abnormal features,using Chebyshev’s theorem estimates the upper of the number of abnormal features.Experimental results show that the proposed method outperforms compared methods in terms of noise immunity and anomaly mining ability.(3)Aiming at the problem of anomaly detection for high-dimensional data,an anomaly detection method based on conformal prediction is proposed.The key thought of the method is that using the conformal predictor calculates conformity scores between unknown data and previous data.Based on the calculated conformity scores,the neural networks predict the class label of the unknown data,and learn compact boundaries separating anomalous and normal classes on conformal regions generated by the conformal predictor.Experiments on smart grid datasets show that the proposed method outperforms competitors in the capabilities both anomaly detection and the learning boundaries.(4)Aiming at the problem of classifying normal data and abnormal data in a high-dimensional space,an anomaly detection method based on conformal transformation with fuzzy is proposed.To explore those hard-to-find sub-regions containing anomalous classes,the ability of nonlinear kernels that adapts to different sub-regions requiring different nonlinearities is enhaced by conformal transformation.To suppress noise interference,using the fuzzy function evaluates the contributions of hypersphere training provided by points.Experimental results show that the proposed method outperforms the competitors in classification ability and noise immunity.