Research On Traffic Reassembly And Distributed Storage Based On DPDK

With the continuous development of Internet technology,the ensuing network attack technology constantly poses threats to the network security situation.The traditional network traffic monitoring method is difficult to meet the current increasing network bandwidth.This article proposes a high-speed real-time traffic reorganization and storage technology scheme based on DPDK framework,which makes up for the shortage of traditional traffic reorganization technology in real time.At the same time,this article uses the method of small file merging and file pre-reading to optimize the read and write performance of traffic file storage.The real-time traffic reorganization scheme and the storage scheme of PCAP files proposed in this article can lay a good foundation for the subsequent traffic audit work.In this article,traffic acquisition and reorganization technology and traffic data storage and other related technologies are studied.Aiming at the performance and efficiency of traffic data reorganization in high-speed network environment,a real-time traffic reorganization scheme based on DPDK framework is proposed.The scheme adopts the independent flow table in each processor core and the design of timeout detection queue,so that the traffic recombination function can be guaranteed in real time in high-speed network.Aiming at the problem that the symmetric RSS algorithm used by DPDK framework cannot satisfy the load balancing among multiple processor cores,this article proposes a set of available load balancing algorithms.By adopting the idea of multiple hashes,the collected network traffic can be evenly distributed to each CPU processing core,which improves the performance of DPDK framework for multi-core processing of packets.Aiming at the problem that too many small traffic files may be generated after network traffic recombination,which will affect the HDFS read and write efficiency,this article proposes a scheme of using FMM to merge small files.By adding the file metadata management between the HDFS Name Node and the client to manage the file index and merge the small files,the scheme can reduce the number of interactions with the Name Node when storing the small files.At the same time,in view of the characteristics that traffic files are usually read in time sequence during file audit,a file pre-reading scheme is designed to improve the reading performance of small files.In this article,the prototype system is designed and implemented on the basis of the above technical scheme.The prototype system can realize the acquisition of high-speed traffic data and the efficient recombination of network traffic.In addition,this article also realizes the persistent function of traffic data based on distributed file system after the completion of traffic data reorganization.This article also carries out experiments and tests on the basis of the prototype system,and the experimental results show that the prototype system can realize the collection of traffic data and real-time traffic data reorganization under the network environment of 10 Gbps,and can persist the traffic data to the disk.In the network environment of 10 Gbps,the efficiency and real-time performance of traffic reorganization have been improved.Compared with the original DPDK scheme,the processing efficiency of traffic reorganization is improved by 12.7%.For the small file merging scheme and the read-ahead cache scheme proposed in this article,compared with the original HDFS,the storage efficiency of small files is improved by 13.9%,and the reading efficiency of small files is improved by 37.3%.