Optimization Techniques Research On Real-time Processing Of Massive Network Streams

Intrusion detection and firewall are general techniques of network security, and the national backbone Internet needs deep detecting of high speed network data and high performance real-time processing of massive network steams. With the development and application of broad band technique, the increasing network bandwidth has exceeded the increasing speed of CPU, which greatly challenges the backbone network intrusion detection system (B-NIDS).The thesis makes the intrusion detection system of high speed wide area network as the background, and studies the related optimization techniques on computer architecture, including how to enhance receiving capability of intelligent nework interface card (NIC), how to design the software structure on multicore platforms and how to guanrantee quality of service. The basic idea of the thesis is to solve the operational difficulties of B-NIDS, and to improve system performance from the aspects of packet receiving rate, throughput, response time and service quality.To enhance the performance of NIC and relieve the performance bottleneck of the host system, the thesis implements a TOE system based on intelligent NIC developed by our lab to receive packets at high speed. In performance analysis, the thesis gives the throughput of the system in various manners of network buffer allocation, and tests the best processing performance of the intelligent NIC. The experimental results show that intelligent NIC improves receiving rate of network packets, even reaches the line speed under some configurations.The thesis starts with characteristic analysis of B-NIDS, focuses on workload and system features that have strong relations with efficient resource management and rapid packet response, and proposes the corresponding optimization methods. For the special traffic characteristics of backbone Internet, the thesis designs proper analysis methods, and gives the optimization ways based on analysis results. The thesis also proposes optimization methods in the view of saving CPU and memory resources. For the further research on the relation between individual streams and their service time, stream samples are extracted from the massive traffic traces and the benchmark is built up.The thesis studies the B-NIDS optimization techniques on multicore and multithreading platforms. Thread-level parallel structure is main trend of current processor development, how to discover the application parallism is the primary question. On the highly parallelled platforms, the optimization research includes analysis of workflow, division of parallel subtasks, workload balance and communication of parallel tasks and so on. After testing the performance bottlenecks of existed B-NIDS and abstracting the features of data flow, the thesis proposes the lock-free synchronization mechanism and the three-level pipelining multithreading model. The performance evaluation is undertaken on a Sun's SPARC T1 server with a processor having 8 cores and 32 simultaneous threads from three aspects of resource utilization, throughput and response time, the scalability is also tested on a Dawning x86 server with the total 8 processors and 16 cores. The experimental results show that the tested metrics are greatly improved and the system scales well on different multicore platforms.The thesis also studies the optimization techniques guided by service quality. The existed system is difficult to ensure the service quality of massive TCP streams, especially under the high speed traffic of backbone Internet, so it provides best-effort service. The improved system preferrentially processes some streams by scheduling policy on several queues, and shortens the service time of those streams having user defined characteristics. The thesis proposes quality guiding techniques based on stream speed prediction and also at the time of overloading. For three system features of having massive streams, heavy workload and processing diversity, the thesis uses the probe-based sampling approach to evaluate the system performance at the policy of preferentially processing some subnet traffic. The experimental results show that the quality of service has greatly improved.