Anomaly Detection System Research Based On Semi-Supervised Learning

In recent years,with the deepening of data science research,anomaly detection has received more extensive attention as an important branch of data science.Anomaly detection is a technique for identifying and mining samples that do not conform to the expected pattern in large-volume samples.It has important significance and value in a series of practical scenarios such as network attack monitoring,structural defect identification,financial fraud detection and medical diagnosis.In some anomaly detection scenarios,the manual labeling cost of abnormal samples and normal samples is high,and it is difficult to use supervised learning algorithms.Although the unsupervised learning algorithm does not need to label samples,the requirements for sample features are usually much higher than the general supervised learning algorithms.How to make full use of a small number of labeled samples and a large number of unlabeled samples is one of the core issues of current anomaly detection.Aiming at this problem,this paper based on decision theory,semisupervised learning theory and deep learning theory,improved the existing semi-supervised learning algorithm,proposed eight semi-supervised deep learning algorithms,and designed an algorithm system to integrate the algorithm.,improved the effect of anomaly detection.The main work of this paper is as follows:(1)The existing semi-supervised learning algorithm based on deep generative model improves the detection effect by generating abnormal samples and normal samples.Because the low quality of the generated samples can significantly reduce the detection effect,the algorithm has higher requirements for the depth generation model.Moreover,the algorithm relies too much on good class prior,which is often difficult to meet in practical applications.In response to these problems,this paper first proposes that the NNPU-GAN algorithm and the NNPU-WAE algorithm can improve the detection effect without relying on the quality of the generated samples,and further propose the S-EM-PN algorithm to reduce the dependence on class prior.(2)Based on the unbiased learning theory,this paper proposes an unbiased semi-supervised learning theory,and proposes the NNPNU algorithm,NNPNU-GAN algorithm and NNPNU-WAE algorithm to further improve the detection effect of the existing semi-supervised learning algorithm based on deep generative model.(3)The semi-supervised learning algorithm is very easy to fall into the local convergence point in practice,which may result in a worse effect than using only the supervised learning algorithm.Aiming at this problem,this paper designs an algorithm system to integrate the algorithm,so that the semisupervised learning algorithm can effectively improve the effect of anomaly detection.Finally,the experiments on the two anomaly detection datasets of KDD99 and NSL-KDD verify the effectiveness of the proposed algorithm,which enables the semi-supervised deep learning algorithm to achieve stable anomaly detection under the simulation of various complex data scenarios.