Research On Some Key Problems Of High-speed Network Traffic Analysis

The increasing network security threats have driven the emergence of various kinds of safety-defense mechanisms, which generally need to analyze network traffic to detect malicious behaviors and harmful information. As the cybercrime methods become various and complex, network traffic analysis should employ stateful analysis based on flow granularity. With the continual growth of network bandwidth, there is great academic significance and practical values on researching the models and the related key problems of high-speed network traffic analysis based on flow granularity.Based on the definition of bit-stream uniformity index, we analyze the impact of the five bit-wise operations, AND, OR, NOT, XOR and SHIFT, on bit uniformity index in theory and experiment. We conclude that NOT does not change the uniformity index, AND and OR usually do not increase the index, XOR increases the index generally and SHIFT increases the index in most cases. Furthermore, we prove that every bit in a bit-stream has an independent impact on the uniformity index of the whole bit-stream if all bits in the bit-stream are pairwise independent and uncorrelated.Targeting the performance requirements of connection record management in high-speed networks, we present an improved efficient hash algorithm PRH (Pseudo Random Hashing)-MTF (Move To Front). We first design a uniform and efficient hash function PRH, based on the analysis of the uniformity indices of network bitstream operation results. To resolve hash collision efficiently, we apply the MTF heuristic to improve the traditional chaining resolution. Selecting packet train model as the packet arrival pattern, we analyze the complexity of the algorithm and deduce its average search length. Finally, we contrast the PRH-MTF algorithm with the traditional sorted hash algorithm on lookup performance and robustness by captured high-speed network traffic and attack simulation.Targeting the performance requirements of state management in high-speed networks, we present an improved robust TCP (Transmission Control Protocol) flow state management mechanism. We first define a specialized state machine for TCP flow as the foundation of state management. To isolate all dirty connections induced by malicious behaviors, we devise the budding connection buffer scheme to improve the performance of connection record management. After that, we classify TCP packet by the essential workflow of state management and further describe the implementation of state management for each TCP packet. Finally, we contrast the improved state management mechanism with the traditional one on efficiency and robustness by captured high-speed network traffic and attack simulation.Targeting the real-time property requirement of high-speed network traffic reassembly, we present an improved fast TCP stream reassembly mechanism. Based on the analysis of network traffic characteristics and TCP connection establishment process, we build several reassembly policies such as recently-accessed-first principle, initiating connection buffer mechanism and payload buffering and reassembling scheme. Then, the reassembly implementation is elaborated in accordance with the policies. Finally, the improved reassembly mechanism is contrasted with the traditional one in terms of efficiency and memory usage by captured high-speed network traffic.Based on the requirements of network user behavior recovery, we devise the design principles of network user behavior recovery system, and present a system prototype. We elaborate the overall framework of the system, discuss the related key techniques such as network traffic capture, TCP stream reassembly and application information recovery, and illustrate the implementation of kernel modules in the system.The research on high-speed network traffic analysis model and its key problems has reached some theoretical and methodological achievements, which are constructive and significant for strengthening network defense capability and fighting network crime.