A Research On Semi-supervised Anomaly Detection

Anomaly Detection is a classic machine learning task that aims to identify abnormal samples in data and has wide applications such as intelligent manufacture,video surveillance and network intrusion detection.In the early 20 th century,with the popularization of machine learning algorithm,there was a great breakthrough in anomaly detection.There were One-Class SVM,Local Outlier Factor and Isolated Forest,etc.,which still perform well today.In recent years,with the popularity of deep learning,anomaly detection methods based on deep neural network have gradually emerged,which have better performance on high-dimensional data compared with traditional machine learning methods.In general,anomaly detection is defined on the unlabeled dataset mixed with normal samples and abnormal samples,in which the normal samples account for the majority.Due to the lack of label,anomaly detection is defined as an unsupervised learning task.In practical application scenarios,we can usually obtain a small number of labeled samples in addition to unlabeled samples.Semi-supervised anomaly detection aims to improve the performance by using a small amount of annotation information.Semi-supervised anomaly detection is closer to the real application scenarios,but it is still in the early stage of development and is a key topic in anomaly detection research.In this context,several problems in semi-supervised anomaly detection are studied.The creative research results of this paper are as follows:1.The data scenario of semi-supervised anomaly detection is formally defined.The characteristics and challenges of this data scenario mainly include: the single data category of anomaly detection leads to the weak discriminant ability of the learned model;The noise in unlabeled samples is easy to lead to the model fitting to the biased input distribution;Small amounts of annotation information should be fully utilized.According to the characteristics of this data scenario,an anomaly detection scheme based on transfer learning is proposed in this paper to enhance the discriminant ability of the model and solve the problem that the anomaly detection method based on deep learning is difficult to benefit from external data.A new loss function is proposed to balance the weight of each sample and enhance the robustness of the noise in the unlabeled sample set while emphasizing the labeled sample.The experiments show that the proposed model is robust to noise and makes good use of labeled samples.The proposed method achieves better results than the existing methods.2.In this paper,the empirical probability of anomaly occurrence is introduced into the anomaly detection model as a priori for the scenario with stable abnormal probability.The category distribution predicted by the model is matched to the prior category distribution by adversarial training.The trained model will consider the prior anomaly probability when predicting the category of the unknown samples,so as to reduce the occurrence of missed detection and false positives in anomaly detection.In this method,the use of labeled samples is no longer limited to minimizing the classification error,and the predicted category distribution is also constrained to conform to the prior category distribution.Experimental results show that the introduction of anomaly occurrence probability can greatly improve performance.The two anomaly detection methods proposed in this paper are both for the semisupervised which is closer to the real application and applicable to different specific situations,so as to provide practical solutions for real-world applications.