Study On High Speed Internet Link Traffic Measurement System Based On Edge

Nowaday, traffic measurement system mainly relies on tcpdump/libpcap-style processing. Design decisions, such as forcing all collection, analysis, and display to be performed on a single machine, as well as implementation decisions, such as single-packet copies from kernel level to user level, mean that tcpdump and libpcap cannot scale to today's Gbps network speeds. Today the most traffic monitors are faced with the following challenge in high-speed network. The paper studies on high-speed network measurement system and it has four parts work as follows: (1) In order to manage the whole network, a novel traffic model is proposed based on the mixed measurement. The core of model has two parts. Firstly, we measure traffic at the edge of network, which the key problem is packet classification algorithm and storing technology. Secondly, we measure route information by injecting probe packet into the network based on the Internet Protocol Measurement protocol (IPMP). In order to probe route information and reduce disturb, we use sampling technology based on packet content. We prove the validity of mixed measurement model and simulate the model. (2) In order to monitor the utilization factor of link bandwidth, a novel traffic monitor technology is proposed based the regression model. The core of model has three parts: First, we inject probe packet pairs into the network based on the Internet Protocol Measurement protocol (IPMP) by Poisson law. Second, we can get the interval of packet pairs by measuring OWD (One Way Delay time). Last, we acquire the regression formula to acquire traffic based on M/M/1 queue model. In order to compute traffic, we get traffic formula by the least square method. We prove the validity of regress analysis method and acquire regress formula. Our computing results show the competing traffic error within about 36% by regress formula. (3) We study packet classification algorithm and propose two kind of classification. One is non-collision hash and jumping table Trie-tree(NHJTTT) algorithm, which is based on Non-Collision Hash Trie-Tree Algorithm and Grid of Tries algorithm. The core of algorithm has three parts: 1) constructing hash function mainly based on destination port and protocol type field so that the hash function can avoid space explosion problem; 2) incorporating two kinds of algorithm, which are the Lakshman and Stiliadis propose a 2-dimensional classification algorithm and the algorithm of Grid of Tries as well as transforming Grid of Tries for the Trie-tree pruned and jumping table in order to reduce space complexity; 3) adding a floor based on Non-Collision Hash algorithm as source port number (or scope).After expanding normally, this don't increase the time complex degree of algorithm because we introduce the jumping table. Space complexity consumed and space requirement are less than those of Non-Collision Hash algorithm. Test results show that the classification rate of NHJTTT algorithm is up to 1 million packets per second and the maximum memory consumed is 8.2MB for 10,000 rules. Two is double-hash(DH) algorithm based on Non-Collision Hash Trie-Tree Algorithm and XOR hash algorithm. Hash algorithm allows us to map an element x in a large set into an element h in a small set through the hash function h=f(x) so that we can quickly find the information once we have determined the small set to search for. The XOR hash algorithm introduces XOR operation to obtain a hash key value. The computation of an XOR hash key value consists of three steps: (1) structuring the non-collision hash function, which is constructed mainly based on destination port and protocol type field so that the hash function usually can avoid space explosion problem; (2) introducing multibit Trie-tree based the key value of XOR hash in order to reduce time complexity; (3) lookup every rule index in order to ensure the validity that we get the final rule index. The test results show that the classification rate of double-hash algorithm is up to 5 million packets per second and the maximum memory consumed is 6MB for 10,000 rules. (4) We develop the software about link traffic measurement system that it may be used kilomega ethernet network. To collect traffic, we require a network interface upon which a copy of all relevant network traffic is available. This can be done using network stack operations, port mirroring, or a tap mechanism. Network stack operations performed by the operating system provide a copy of data to the libpcap program running on a given host. Libpcap may perform additional processing before passing it on to tcpdump for display. Port or interface mirroring is a technique by which the traffic from one or more interfaces on a network switch (the mirrored interfaces) is copied to another port (the mirroring interface). In theory, this provides a mechanism to transparently observe traffic passing over the mirrored interfaces by observing traffic over the mirroring interface. A tapmechanism is a piece of hardware that takes a single network input and duplicates it to transparently produce two identical outputs. This can be thought of as a splitter or a switch performing half-duplex port mirroring. This hardware works at the physical level by splitting a physical signal and possibly enhancing it. Flow information can be submitted to different network analysis applications, which can get data from flow storage server for long term analysis, or can get flow data use IPFIX protocol for real-time monitoring, such as performance evaluation, workload characterization, protocol debugging, network troubleshooting and usage-based billing etc.. A careful study of the use of software measurement techniques shows that it is possible to significantly improve them. However, the process is far from trivial. Most of our work has taken place in the context of the Linux operating system where the source of the code is available. Without this it would be impossible to understand the operating system effects introduced into the measurement process let alone do anything about them. This Linux kernel is still a fully functional operating system at this point, stripped only of unused code for a given machine. The next step is to further remove services and functionality unused by the monitor that would reduce performance. For the uniprocessor hardware on which we run the monitor, the only required changes were replacing the init() function with a call to the monitor code removing calls to the proc filesystem, and removing hardcoded hardware probes for non-existent devices. Other implementations may wish to turn off virtual memory handling, remove printk statements, and so forth. Now we are left with an absolutely minimal kernel to manage low-level hardware tasks and provide a useful Application Programming Interface (API), which we use to actually program the monitor. The test results show that the collecting packet rate of our designing traffic measurement system is up to 0.43 million packets per second, but the collecting packet rate of Tcpdump measurement system is only up to 0.2 million packets per second.