The Research Of Content Filtering System Based On Massive Data

With the rapid development of the Internet and the increasing penetration of network applications, the amount of data transmitted on the network is accompanied by explosive growth. People to enjoy the network brings a lot of convenience while also facing unprecedented security threats, such as malicious viruses, obscene, reactionary content, network crime increasingly serious, the daily life of the people are faced with a serious threat to national security, Therefore, the effective monitoring and management of Internet content, combat cyber criminal acts without delay. Data content filtering system on such a network data content monitoring the professional system.Data content filtering system and its key technology has become the focus of the current field of network security, the paper proposes a filter filtration system model based on the massive data packet capture, load balancing algorithms, application protocols restore discussed and matching algorithm for the in-depth study. The main contents of the following four aspects:(1) Filtration system structure model based on the contents of the huge amounts of data. The model of the system divided by function and be able to analyze and filter the massive network data content quickly in the case of low packet loss rate;(2) Far exceeds the processing capability of the processor core network for the current data traffic, and a new load balancing algorithm. Algorithm to the huge amounts of data in the core network according to certain rules distributed to a host of multiple data analysis and processing, distribution process to ensure that the same session data to the same reduction machine for processing, in order to avoid packet loss happens, to ensure that the data integrity;(3) Application layer protocol for this network hosted a wide range of difficult to identify the problem, propose an application layer protocol analysis and restore the program. The program based on the port number of the load, as well as popular comprehensive way to identify the application layer protocol. The algorithm can quickly identify the application protocol, and the accuracy rate of more than99%; (4) Proposed a new pattern matching algorithm. In order to deal with the threat posed by the increasing amount of network data for network security, the paper proposes a new pattern matching algorithm and its application to the mass data filtering system. Improved algorithm significantly reduces the matching time, improve the efficiency of the implementation of the filtering system of massive data;(5) Presents a network packet capture and processing solutions based on the Linux system kernel mode, and explain in detail to the TCP/IP protocol data reduction module.