Study On Dynamic Cost-sensitive Learning And Deep Belief Network For Anomaly Detection

With the development of big data,the amount of data is expanding,and a steady flow of data is likely to include abnormal data,fraudulent data,etc.,which have serious impacts.These data not only affects people's lives and work,but also affects the safety of society,even affects the security of our country.Therefore,detecting abnormal data and fraudulent data has becoming an urgent task.These data is imbalanced data.The imbalanced data is that the number of different categories of data differs a lot.These abnormal data requires us to learn their characteristics and make effective predictions.The existing imbalanced data processing methods are based on data and based on algorithm.The disadvantage of the method which based on data is that when the amount of data increases,the time complexity for processing will increase exponentially;the method which based on algorithm includes machine learning methods,deep learning methods,and integrated learning methods.With the development of deep learning,neural network technology has also gradually developed.In anomaly detection or fraud detection,neural network technology has the characteristics of high recognition rate and good stability because it can simulate the basic characteristics of human brain.In addition,in consideration of the characteristics of little labeled data of abnormal data and high cost of manual labeling,the concept of semi-supervised learning is introduced.Semi-supervised learning improves the performance of classifier by adding unlabeled data to the training of labeled data.Based on these,this paper makes the following researches on anomaly detection:(1)Investigate the research situation and the development tendency of anomaly detection technology and summarize the research results of predecessors.Study the theory and methods of the existing main anomaly detection algorithms,and summarize the shortcomings of these methods.(2)A cost-sensitive deep belief network classification method is proposed to solve the problem of imbalance in anomaly detection,and the disadvantages of traditional DBN can not effectively solve the imbalanced data,thus introducing cost-sensitive learning technology.In addition,the misclassification cost of the softmax layer makes the DBN sensitive to the classification of imbalanced data and improves the accuracy of classification.(3)For the cost-sensitive deep belief network method proposed in(2)and the disadvantage of the manual setting of misclassification cost,the method of dynamic cost-sensitive deep belief network is proposed.The experiment is carried out on a number of public imbalanced datasets.ROC curve is used as the evaluation indicators of the model.The model is compared with the traditional non-equilibrium processing algorithms and obtains good experimental results.(4)An algorithm combining semi-supervised learning and dynamic cost-sensitive deep belief network classification is proposed.And the algorithm is applied to the fraud dataset.By experimenting on the dataset and measuring with evaluation indicators,good results can be obtained.