Design And Implementation Of Client For Distributed Data Collection System

With the rapid development of Internet and network applications,network management is becoming more and more important. Network packet collection and analysis is a basic technology used in network management, is an effective network management methods, can be used for network traffic monitoring and network load testing. Through analyzing the network data packet, the network status can be known in real time, such as network response time, packet loss rate, network load,network bottleneck, network intrusion and illegal network packet contents. It has the important meaning to the network management. With the increase of network bandwidth and the popularity of distributed applications, the traditional data packet collection software has been difficult to deal with distributed clusters and high-speed network under the pressure of large-scale data collection.In order to solve the shortcomings of the existing data collection tools, this paper presents the design and implementation of a distributed data acquisition system based on Libpcap and PF_RING. A multithreaded concurrency model is proposed to ensure that the acquisition client can collect, filter and execute the network data packets in parallel on the multi-card network card during the acquisition process. At the same time,the lock-free ring buffer technology and the processor affinity technology are used to optimize the acquisition performance and reduce the packet loss rate. This paper proposes an important auxiliary function for the client runtime: designing multi-thread asynchronous log writing algorithm, using NTP time synchronization service and designing heartbeat protocol to ensure the stable operation of client.This paper first introduces the background of the distributed data acquisition system, combs the main work content of the client and analyzes the requirements of the tool. After a series of investigation and analysis work, it solves the problem of high-efficient data collection technology Then, the whole architecture of the client is designed,and the modules and the key data structures are designed in detail. Finally, the paper introduces the design and implementation of the acquisition client A complete functional test and performance test are carried out to verify the completeness and validity of the acquisition tool. In this paper, based on Libpcap and PR_RING packet capture packet capture portal client,provides an easy to use configuration file entry, and finally achieve a stable and efficient cluster node data collection in parallel, to provide users with a variety of acquisition mode,Facilitates the network management user to carry on the investigation and the retrospective to the network data.