Research On Anomaly Detection Based On Fuzzy SVD And XGBoost

In the era of big data,the traditional abnormal traffic detection adopts the static rule matching method,which can no longer satisfy the changeable and complex network environment.Facing a larger and more complex network environment,the characteristics of network security data are no longer just a "black and white" data description.Massive complex data has the characteristics of "volatility" and "uncertainty" of information generation.Fuzzy set theory is widely applied to the constraint conditions of data feature extraction.It makes the extracted features more diverse and comprehensive,and makes the subsequent data analysis and prediction more comprehensive and specific.Therefore,this paper adopts SVD(singular value decomposition)and XGBoost(limit gradient promotion tree)methods in machine learning to detect anomalies.Anomaly detection is divided into data preprocessing and data analysis.In the data preprocessing stage,a manifold constraint feature selection method based on fuzzy SVD is proposed.In the data analysis part,a nonlinear weighted XGBoost anomaly detection method is proposed.Network traffic data due to its complex internal structure.The volume of data,the variety of data,and the existence of redundant data.In the process of data mining and information processing,there will be a huge amount of computation,which will reduce the efficiency of analysis and processing.Therefore,it is necessary to carry out dimensionality reduction processing for high-dimensional network data.Based on this,this paper proposes a manifold constraint feature selection method based on fuzzy SVD,which mainly solves the dimension reduction problem of high-dimensional data of network security.By combining the intuitionistic hesitation fuzzy set with SVD,the ordered optimal feature subset can be solved quickly.On this basis,supervised dimensionality reduction can be carried out.After dimensionality reduction projection,the effect of clustering of similar data and dispersing of different kinds of data can be realized.Aiming at solving the problem of low classification accuracy of anomaly detection model.In this paper,we consider that the characteristic distribution rate of some network abnormal data is very low,but it can provide great contribution in data analysis.A nonlinear weighted XGBoost anomaly detection method is proposed.In this method,the obtained ordered optimal feature subset is nonlinear weighted into the design of the objective function.The nonlinear weighted loss function is established to balance the analytic contribution weights among the features with large drop sample proportions.In addition,in order to reduce computation and improve computation speed,a portion of the training set is randomly selected in the decision tree construction of XGBoost.Finally,the simulation experiment proves that the manifold constraint feature selection method based on fuzzy SVD proposed in this paper greatly speeds up the calculation speed of feature extraction compared with other traditional feature extraction methods.The proposed nonlinear weighted XGBoost anomaly detection method is effective in improving the anomaly detection efficiency.Through the experimental analysis of data preprocessing stage and data analysis,it is proved that the method proposed in this paper can effectively improve the accuracy of anomaly detection.