Research On Network Abnormal Traffic Detection Method Based On LightGBM

The rapid growth of the current network size and the diversity of network attacks have led to increasingly serious security problems facing the network.Network anomaly traffic detection can identify attacks and issue alerts in a timely manner,which is of great importance to ensure the safe operation of the network.However,due to the huge volume of data,high-dimensional redundancy of features and data imbalance in network traffic,traditional machine learning algorithms are no longer sufficient for current detection needs.In order to perform network anomaly traffic detection efficiently and accurately,this thesis proposes a LightGBM-based network anomaly traffic detection method based on the LightGBM algorithm’s ability to handle massive amounts of data accurately and in real time.The main research work is summarized as follows.(1)Propose a BGWO-LightGBM-based network anomaly traffic detection method to improve the classification accuracy.First,to address the problems of highdimensional redundancy of network traffic features,which can affect detection accuracy and lead to long computation time,the Binary Grey Wolf Optimizer(BGWO)is used for feature selection,taking advantage of BGWO’s ease of implementation,fast convergence speed and superiority-seeking capability to select the optimal feature subset.Then,the Adaptive Genetic Algorithm(AGA)was used to perform hyperparameter search for LightGBM to address the problem that the classification effect of LightGBM is heavily influenced by hyperparameters and the tuning of parameters is tedious.Experimental validation on the UNSW-NB15 dataset and CICIDS2017 dataset shows that the BGWO feature selection method outperforms the other three heuristic feature selection methods in terms of finding the best feature subset,effectively reducing the feature dimension and eliminating redundant features,improving accuracy and reducing computation time.AGA can optimize the LightGBM hyperparameters and further improve the classification accuracy.The method is an accurate and efficient method for network anomaly traffic detection.(2)A KSIFL-based network anomaly traffic detection method is proposed to address the problem of low detection rate of a few classes due to data imbalance existing in network traffic.KSIFL is a combination of data-level and algorithm-level imbalance processing method.First,a few classes are oversampled at the data level using the KSI sampling algorithm using K-Means SMOTE,Isolation Forest(IF)algorithm to remove noise;Then combined with the algorithm-level FL-LightGBM,which uses Focal Loss to improve the loss function of LightGBM and enhance the learning of hard-to-classify samples during training.KSIFL reduces the degree of data imbalance and improves the prediction accuracy for difficult-to-classify samples.The experimental results show that KSIFL can effectively improve the minority class detection rate and outperforms RUS,SMOTE and KSI.(3)Based on KSIFL,a Voting-integrated network anomaly traffic detection method based on Voting is proposed to further improve the detection rate of minority class anomalous traffic in multi-classification tasks,to avoid limited improvement of minority class detection rate by a single imbalance processing method,and to improve the generalization of the method.Soft-Voting integration of several imbalance processing methods with different advantages including KSIFL,KSI+EasyEnsembleXGB,Balanced-RF and WL-LightGBM,fusing the advantages of each method to further improve the detection rate of a few classes in a multi-classification task,while adding LightGBM to ensure accuracy.Experimental results show that the Voting integration results in better minority class detection rates than any single imbalance processing method and better overall accuracy than all three imbalance processing methods except KSIFL,with improved minority class detection rates and generalisation of imbalance processing methods in multi-classification tasks.