Design And Implementation Of Distributed Connection Log Storage And Retrieval System Aimed At IP Address Calibration

With the rapid development of Internet, network applications and network traffic continues to grow, to human social life, economic convenience, but has also brought great challenges to network management and network security has brought great challenges. Connection log is generated by a network session, which is a good description of the network from the session level. So how to provide reliable storage for high-speed, long-term connection logs, how to discovery IP social attributes accurately based on connection logs, plays a crucial role in network security, network management and network planning.Existing storage solutions for connection logs are not well balanced reception, storage and retrieval, so they receive limit portability. Connection logs generated by multiple backbones per second can reach millions of levels, which makes traditional centralized storage solutions increasingly unable to meet the demand. Then some distributed framework provides scalable storage performance, but the storage engine of these framwork mostly based on the traditional relational database, with limited storage capability. In this paper, we achieve connection log storage system supports distributed high-speed storage, high-speed query through the depth study of storage solutions. In addition, the depth study on the traditional IP social attributes discovery found that traditional port-based and behavioral characteristics based solutions with lower resolution. In this paper, a detailed analysis of the connection logs, in order to achieve a more accurate calibration of the IP social attributes. The main content of this paper can be summarized as follows:(1) Proposed a new high-speed connection log receive framework frame:DPIO (Driect Packet I/O). Although tradition Socket API can be relatively simple to achieve connection logs receivers, but its performance is not high. The new network driver netmap, can solve this problem, but netmap need to maintain a separate network card drivers, are more difficult to implement and maintain. This paper proposes a new framework in the connection logs receive, the experimental results show that DPIO can solve the problem of receiving low rate of Socket API, but also to avoid the complexity of the netmap.(2) Designed and implemented distributed connection log storage system supported high-speed storage and quick retrieval:DCLStore (A Distributed Connection Log Storage System Supports High-speed storage and Fast Retrieval). DCLStore is able to provide high-speed storage and high-speed retrieval capabilities, and provide scalable storage space by dynamically adding storage nodes. The experimental result shows that the system can process about20million connection logs per second. DCLStore can provide40times higher than the single point of storage systems under the same storage capacity corresponding to the speed of the query.(3) Proposed new IP index:IP activity. The traditional port-based and behavioral characteristics based solutions of the IP social attributes though simple, but accuracy is not high. This paper proposes a new IP index:IP activity. Then, the paper uses IP network infrastructure to indicate the process of the connnection logs is right. Finally, the use of open source tools for network-wide IP activity is calculated, and study its impact on the calibration of the IP social attributes. Experimental results show that, IP activity has a great impact on the calibration results based on the IP port and behavioral characteristics.