The Research Of Distributed Network Packets Indexing Storage System

With the rapid development of network technology,the global network data flow has increased rapidly in recent years and may increase more quickly in the future.In today's big data era,the value of flow data is particularly important.Enterprise and service provider networks,financial institutions,the data center with high security requirements pay more and more attention to some tools,these tools can archive network traffic information for post hoc analysis.However,archived network traffic information is very large,how to organize the big data for security analysis is a very important matter and a big challenge.Network data needs a lot of storage space,it is a problem that how to solve such a large storage requirements.In this paper,we think about using distributed structure,making a large number of cheap computers together to constitute a huge storage space.To satisfy the large storage requirements just only solves the problem on the one hand,on the other hand,there is another problem that how to organize the huge data to make users can easily access it.In this paper,we decide to adopt the index technology for data organization,with the index database as well as the directory of the dictionary,it can greatly accelerate the procession of accessing to data.Based on the multidimension of network data,in this paper we use the bitmap index.According to the above,this paper actualizes a fast distributed network data packets indexing system,it is based on bitmap index.In this system,there are two important concepts of blobs and tags.The two concepts are responsible for data distribution,replication,persistence,addressing and access.Using this system we can fetch the important information in the head of the captured data packets to establishing the corresponding bitmap index;memory-mapped database is adopted to store index information;the whole system is based on the Map Reduce programming model;the storage information can be copied to different servers throughout the distributed system to enhance the reliability of the system;the servers in the system can freely join in or leave,so that the system has good scalability;all the servers in the system can operate insertion and query independently to support multiple users well.We implement our system using some relatively low configuration computers,evaluate this system based on the captured network packets from the real network.We get a good results,insertion rate of a single computer is about 30,000 records per second,and the query rate is about 5,000,000 records per second;when N computers execute parallelly,the total efficiency is N times of the above.