Real-Time Elephant Flow Detection Mechanism Design And Implementation In High-speed Network

In recent years,with the deepening of the interconnection of all things,more and more networked devices are closely interacting through the Internet,creating a continuous increase in network data traffic in high-speed networks,which not only makes the network environment increasingly complicated,but also makes the network security situation more complex.High-speed network traffic measurement is an important function to obtain network performance and support network management.Its main goal is to measure network flow-level traffic information and provide decision-making basis for key network functions such as load balancing and attack detection.Therefore,in recent years,network traffic measurement technology has received extensive attention from academia and industry.The main challenge in high-speed network traffic measurement is the mismatch between severely constrained on-chip storage resources and massive high-speed data traffic.Traffic measurement modules need to record network flow information using high-speed but very spaceconstrained high-speed on-chip storage.Moreover,the network traffic distribution is a typical long-tail distribution.Most of the network flows are small-scale network flows,and only a few elephant flows occupy more network resources.Allocating the same computing and storage resources to these network flows with huge differences in size will cause a lot of wasted resources and reduce the overall system performance.Therefore,how to reasonably allocate and use limited on-chip storage resources,efficiently detect elephant flows with high network resource occupancy,and accurately record the information of elephant flows,so as to provide a reliable decision-making basis for fine-grained network management applications,is an important task in network traffic measurement.This thesis mainly concentrates on the elephant flow detection in network traffic measurement,aiming to improve the accuracy of elephant flow detection and reduce the on-chip memory usage as the optimization goal,and design the corresponding elephant flow detection mechanism and traffic information storage mechanism to effectively measure the elephant flow in the network.The main contributions of our work are as follows:(1)Top-k elephant flow detection mechanism based on dual-mode counters.Aiming at solving the problem of low space utilization of equal-length counters,this thesis first designs a dual-mode counter data structure,which only uses a small amount of storage resources to provide a counting range sufficient to record the frequency of elephant flows,while ensuring the accuracy of frequencies of mouse flows;The research combines the proposed dual-mode counter and dynamic exponential decay mechanism to design an efficient Top-k elephant flow detection algorithm,which provides accurate elephant flow detection results and frequency measurement data.(2)An heavy hitter detection mechanism based on IP sharing.Aiming at solving the problem of consuming a lot of space when recording IP addresses,this study proposes an IP space sharing mechanism.By recording multiple IP addresses in the same data structure,it can record more flow information using less on-chip storage resources.In addition,this thesis utilizes the proposed IP sharing mechanism to design an efficient heavy hitter detection algorithm,providing accurate large flow detection results and frequency measurement data.(3)An elephant flow detection prototype system based on FPGA programmable device.This research implements a elephant flow detection prototype system on NetFPGA with the help of its programmable logic array.This thesis verifies the high reliability of the large flow detection prototype system designed in this study by implementing multiple communication methods for data transmission and interaction with the server.To sum up,we analyze the shortcomings of the existing elephant flow detection mechanism in detection accuracy and storage efficiency,and design a dual-mode counter and IP sharing mechanism for the problem of network flow information storage.We design and implement the Efficient and accurate Top-k elephant flow detection mechanism and heavy hitter detection mechanism,and finally build a large flow detection prototype system based on programmable network hardware equipment,which provides important theoretical guidance and technology for large flow detection in network traffic measurement support.