Real Time Traffic Analysis System Based On DPDK

With the popularity of mobile Internet and the rapid development of 5G the scale of data on the network grows exponentially,and the value of data is constantly explored and utilized.As the infrastructure of data transmission and storage,the data center is carrying these complex network data.With the increasing scale of data,huge network data,and high-speed network transmission,how to obtain,analyze and store network traffic in real-time is a challenge that network traffic analysis must face.How to collect and analyze the traffic of the whole data center efficiently without affecting the normal business data is a hot spot.At present,the traditional real-time traffic analysis system is based on the traditional protocol stack,but in the high-speed network environment,the traditional real-time traffic analysis system will have a very high delay rate and packet loss rate.Aiming at the high-speed network environment at the exit of the data center,based on this we design an engine system based on DPDK and DPI which can identify high-speed traffic to adapt to the high-speed network environment.In this way,we can build and analyze clusters.We use Kafka's distributed message queue technology and Spark Streaming's real-time computing framework to analyze and process the data collected and processed by the cluster,and uses Druid's big data query engine to store the data after real-time analysis and processing.Finally,the popular Grafana is used to visualize the data according to the demand.All in all,our system achieves flexibility,scalability,and stability through modular design and the availability of real-time monitoring and alarm functions.The test results of the system in a convergence room show that the average single core processing capacity of the system reaches more than 9.25Gb/s when the peak traffic is 300Gb/s,and the throughput of the system reaches 99.7%.In contrast,based on the traditional protocol stack,the throughput of Libpcap is less than 1.2%.Which means the real-time processing capacity of our system processing capacity of the system is enough to cover the network traffic at the exit of the whole data center.The online system has already helped the relevant personnel to carry out much abnormal traffic alarm monitoring and traffic scheduling and equipment cutting,reducing the data center network management and operation cost,basically meeting all the needs of the actual production environment at this stage.