Network Data Measurement And Anomaly Detection Based On Low Rank Decomposition

With the rapid development of computer networks,the scale of the network is get-ting larger and larger,which brings many challenges to network measurement and network anomaly detection.On the one hand,network performance data is the basis of network anomaly detection,and the key to obtaining network performance data lies in network measurement technology.Traditional network measurement technology often targets the actual network with a node size of n,and the cost of obtaining the performance data of the entire network usually requires O(n~2).When the network size is large,the network is measured by existing measurement methods.The measurement is obviously impractical,and the network performance data has continuity.Continuous measurement of the entire network performance data not only requires a large communication cost,but also cannot meet the timeliness.On the other hand,with the continuous expansion of network scale and the continuous deepening of network applications,network attacks are becoming more and more harmful,threatening the normal operation of the network.In severe cases,large-scale network attacks(such as distributed denial of service attacks DDo S),large-scale worm Worms outbreak,etc.)will bring catastrophic consequences to the network,and the tradi-tional network anomaly detection technology has accuracy and speed problems.In order to reduce the cost of network measurement to adapt to large-scale networks;accelerate the speed of network anomaly detection to meet the real-time nature of network management;improve the accuracy of network anomaly detection to make network management more efficient.This article solves the above challenges through in-depth research on low-rank decomposition techniques,and obtains the following main research results:1.Network measurement scheduling algorithm based on bipartite graph matrix com-pletion modelingIn order to reduce the cost of network measurement,this paper proposes a network measurement scheduling algorithm based on bipartite graph matrix completion model.The remaining unmeasured data can be inferred from a few measured data.Unlike traditional measurement methods,the proposed method models network data into a matrix model,and proposes a matrix completion technique to achieve speculation tasks.In order to reduce the redundant measurement,to determine the measurement position and the measurement stop condition,it is proposed to model the matrix completion problem with a bipartite graph,and based on the bipartite graph model,two measurement scheduling strategies are proposed to ensure that the degree of each node of the bipartite graph is greater than It is equal to the rank of the network data matrix to ensure that the network performance data can accurately infer the conditions;and use the cosine angle to select the non-pathological linear equation system to confirm the measurement position.In order to further reduce the measurement cost,it is proposed to convert the measurement cost of each measurement position into a bipartite graph edge weight and add it to the decision of measurement scheduling.Exper-iments show that the proposed algorithm can achieve low-cost and high-precision network measurement.2.Network anomaly detection algorithm based on matrix decomposition and reuse accelerationBecause existing anomaly detection algorithms usually require a high computational cost,they are not suitable for large-scale network data processing.In order to make the anomaly detection algorithm more suitable for large-scale network data processing,this paper proposes a network anomaly detection algorithm based on matrix factorization and reuse acceleration.Firstly,we conduct an in-depth study on the anomaly detection algorithm based on matrix recovery,and find that the reason for the high computational cost of the matrix recovery algorithm is the singular value decomposition process involved in the low rank recovery process.To solve this problem,this paper proposes a large-scale network anomaly detection method based on a lightweight matrix recovery algorithm.Through the previous experiments on the real data set,it was found that the abnormal location located by the matrix recovery algorithm can be quickly determined without change.Based on this,this paper reduces the computational cost of the current iteration process by reusing the results of the singular value decomposition in the previous iteration process to achieve rapid anomaly detection,and makes the traditional anomaly detection algorithm more suitable for large-scale network data deal with.Experiments show that the proposed algorithm can not only ensure the abnormal positioning accuracy of the matrix recovery algorithm,but also greatly reduce the computational overhead.3.Network anomaly detection algorithm based on continuous truncated high-order tensor decompositionSince the existing network anomaly detection algorithms usually model the data as a matrix model,which makes it unable to make full use of the high-order structured informa-tion inside the data and lose the detection accuracy,for this reason,this paper proposes a continuous truncation based on high-order Zhang Network decomposition anomaly detec-tion algorithm.The algorithm learns the high-order structured information contained in the data by modeling the network performance data as a third-order tensor model.In addition,the low-rank decomposition method based on tensor model is too expensive to calculate and is not suitable for large-scale network data processing.Therefore,this paper proposes to use a continuous truncated high-order singular value decomposition method to reduce the computational cost of the anomaly detection algorithm,and achieve a further reduction in the computational cost by scheduling the truncation order.In addition,this paper pro-poses to use non-relaxed constraints for the anomaly detection problem,by transforming the data separation method based on the tensor model into two sub-problems and iteratively solving,in order to improve the accuracy of network anomaly detection and localization.Experiments show that the proposed algorithm is more accurate than the matrix model-based algorithm and convex relaxation algorithm.At the same time,it is cheaper than the traditional tensor decomposition algorithm and is more suitable for large-scale network data processing.4.Online network anomaly detection algorithm based on bidirectional two-dimensional PCATraditional network anomaly detection is often implemented by data separation meth-ods,and the calculation cost of these methods is too high and requires iteration,so it is not suitable for online real-time alarms.To this end,this paper proposes an online anoma-ly detection algorithm based on a two-dimensional two-dimensional principal component analysis method.Different from the traditional PCA method,in order to fully mine the characteristics of network data,this paper proposes a two-dimensional two-dimensional principal component analysis method to determine whether the new data is abnormal da-ta.In addition,since network management usually requires online processing of streaming data,this paper proposes an incremental update method to update the principal component vector to quickly update the principal component direction of the overall data,thereby re-alizing online network abnormal data alarm.Finally,a method of data augmentation is proposed to strengthen the impact of newcoming data on the overall data,thereby ensuring the accuracy of anomaly detection.Experiments show that the proposed algorithm can fully satisfy the real-time performance of online operation and maintenance of the network,and at the same time maintain a high accuracy of abnormal alarms.5.Network anomaly localization algorithm based on tensor decomposition and reuse under sliding window modelFor actual network operation and maintenance tasks,it is usually necessary to locate the abnormal network location in real time to ensure network reliability.To this end,this paper models the network monitoring data as a sliding window model to ensure the validity of the data while reducing the size of the processed data.In addition,the traditional anomaly localization method is usually designed for offline data on the network,usually separating the network data into normal data and abnormal data.This kind of algorithm not only has a high calculation cost,but also requires high storage requirements,and cannot meet the network control center.Online demand.Therefore,this paper proposes an online tensor recovery algorithm.For the first iteration and subsequent iterations after the arrival of new time data,different CP decomposition algorithms are designed.By fully reusing the factor matrix obtained at the previous time,it is the same as the previous iteration In the process,the factor matrix is sought to reduce the calculation cost,and the online network abnormal location is successfully achieved.Experiments show that the proposed algorithm can not only ensure the accuracy of abnormal location similar to the offline algorithm,but also greatly reduce the computational cost to meet the real-time requirements.6.Network anomaly detection algorithm based on manifold learningIn addition to the multivariate linear relationship,the network monitoring data tensor also has a nonlinear relationship.For example,the network traffic data at each moment reflects the traffic at different moments under the same network topology.The direction and size of these flows are determined by many factors,such as working days,holidays,working hours,leisure time and other factors.In order to be able to integrate and use the nonlinear relationship in the network monitoring data to improve the detection accuracy,this paper proposes a network anomaly detection algorithm based on manifold learning,adding nonlinear constraints to the traditional tensor recovery algorithm to learn the internal non-linear data Linear features.It also proposes a method based on locally sensitive hashing to perform data clustering,which is different from the traditional KNN clustering method.This method can avoid the error caused by the fixed K neighbors in KNN,and at the same time,can reduce the clustering process.Calculate the cost.Experiments show that the proposed algorithm can achieve higher anomaly detection accuracy than traditional linear algorithm models.