Research On Indexing And Querying Network Packets Based On Wavelet Tree

In recent years,with the fast development of internet network and network application,network monitoring and network security become more and more important.For them,long-term historical analysis of captured network traffic is a topic of great interest.It can realize the effective network monitoring and locate the faults in the network to ensure the security by analyzing the network packets.The key requirement to achieve the above functions is how to retrieve the matched network packets from the mass of network packets quickly and accurately.For this purpose,the establishment of effective indexing is very important.The volume of network packet traces becomes larger and larger,and much more complex than before.The reason is that the speed and size of network increasingly expanded.Thus,massive storage space and efficient software tools are necessary to deal with big traces including a number of packets and complex trace patterns.However,traditional tools such as Tcpdump and Wireshark are inefficient to handle very large packet traces.In this paper,we have studied the technology of network packets indexing,there are problems in the storage overhead of index files and the indexing and querying performance improvements.This paper mainly improves the indexing performance,we propose a new efficient indexing and querying method for network packets,and we design and implement a system called Pi,the specific research work is as follows:At First,we use a new data structure named Wavelet Tree for storing the index files,and combine RRR with variable length prefix-free encoding to improve Wavelet Tree,so the indexing takes less time and space overhead.Secondly,we use the improved Wavelet Tree and bitmap compression technique to perform packet queries,the indexing and querying performance of Pi is good.Pi indexes rapidly and it can provide a powerful feature set for queries,which contains the following queries:simple and complex point values queries,simple and complex range values queries,exact query of attributes for maximum and minimum queries,values of query attributes are arranged in ascending or descending.It allows quick access and execute queries on the indexing files.The experiment platform is the machine with HDD and SSD,the results show that Pi reduces the index data size down about 2%of the original pcap files.Both space and time overhead for indexing are less,what's more,time overhead is particularly more less on the machine with SSD.In terms of time for the index data size and packets query time,the performance of Pi is good.Pi shows remarkable performance enhancements in comparison with traditional tools such as Tcpdump and Wireshark,and most recent tools such as Pcap-index,CPPIP and Pcaplndex in terms of index data size and packet extraction time.