Deep Learning Based High-Speed Network Traffic Measurement Research

As the foundation of obtaining network performance and monitoring abnormal behavior,high-speed network traffic measurement provides critical decision-making support for crucial network functions,including load balancing,congestion control,and routing.With the development of next-generation information technologies such as 5G networks and cloud computing,network data is increasing rapidly,requiring lots of high-speed memory and computation resources to process network traffic data in real time.However,the high-speed memory resource that can match the processing requirements of flow rate is extremely limited,which cannot meet the need for direct storage of flow information,and poses a severe challenge for high-precision network traffic measurement.Existing solutions can be divided into two categories:sampling-based measurement methods and sketching-based measurement methods.Sampling-based algorithms can effectively reduce the amount of data that needs to be processed by selecting a subset of elements.However,due to uneven network traffic distribution,sampling-based algorithms cannot guarantee the measurement accuracy of small flows and even completely omit their information.Sketching-based algorithms directly utilize the network processor chip’s on-chip memory to store traffic information,which can efficiently process all network packets.However,different flows often need to share the same memory due to the compact memory space,resulting in interference among flows,seriously affecting the accuracy of traffic measurement.This dissertation proposes a series of high-speed network traffic measurement methods based on deep learning to improve and complement existing researches,which profoundly integrates deep learning technologies with high-speed network traffic measurement,significantly improving the performance of high-speed network traffic measurement.The main research contents of this dissertation are as follows:(1)Per-flow cardinality estimation algorithm in high-speed networks based on sketch and neural network.To address the problem that cardinality estimation algorithms based on memory sharing will introduce a lot of noise during the estimation process,we propose a neural network-based per-flow cardinality estimation algorithm.The proposed algorithm improves the existing packet processing and memory update rule,developing an efficient encoding method to reduce noise as much as possible,using the neural network model to learn potential patterns from the encoded data to improve the performance of per-flow cardinality estimation,which reaches higher measurement accuracy and lower memory overhead.(2)Per-flow cardinality estimation algorithm in high-speed networks based on sampling and residual network.We propose an efficient sampling-based and learning-based per-flow cardinality estimation algorithm to overcome the shortcoming that sketching-based algorithms cannot be deployed on a large scale in today’s network environment.The designed algorithm adopts the packet-level random sampling strategy and proposes an efficient data encoding mechanism to process the sampled element information.Then,we propose a lightweight residual network model to fit the mapping relationship between the encoded data of each flow and its cardinality,which realizes the high-precision per-flow cardinality estimation on general-purpose CPU equipment.(3)Residual network-based multi-objective estimation algorithm.We propose a residual network model-based multi-objective estimation framework for the problem that existing multi-objective estimation algorithms cannot solve the measurement of various fine-grained flow-level metrics.To begin with,we present an efficient encoding scheme to encode each packet into a virtual element containing flow size/cardinality information.Then,we design a virtual element-level non-repeated sampling technology based on a bloom filter.We further develop a lightweight residual network model to learn the mapping relationship between sampled data and various measurement metrics,which realizes real-time and high-precision measurements of multiple flow-level metrics.The above research contents have improved and made up for the shortcomings of existing high-speed network traffic measurement technologies from different aspects,which provides new solutions for achieving high-precision network traffic measurement.This dissertation conducts simulation experiments on the proposed algorithms based on real-world network data sets.The experimental results show that the proposed algorithm has higher measurement accuracy and lower memory overhead than the latest research results.