Research And Implementation Of Detecting Duplications Technologies In Network Security Monitoring

With the rapid development of computer technology and increasingly extensive network applications, network security monitoring plays an increasingly important role in some fields, such as network maintenance, network security and network user behavior analysis. However, the requirement of efficiently dealing with massive network security monitoring data is in conflict with limited resources, such as computation, storage and network bandwidth. One of the most important challenges faced in network security monitoring system is how to process the unlimited rapid continuous network security monitoring data by appropriate technology under the circumstances of limited computation and storage resources. Detecting duplications in data stream is an important link in stream data processing. On the one hand, analyzing duplications which have already been detected provides support for follow-up network security monitoring applications. On the other hand, eliminating duplications is helpful for effectively network transmitting and efficiently online analyzing and processing.Based on the background of network security monitoring, this paper focuses on the problem in processing network security monitoring data, and researches DSMS and detecting duplications technologies in data stream. The main contribution is concluded as follows.Firstly, we research a lot of public DSMSs and analyze their application fields, architecture, query language, components and characteristics. According to the demand of network security monitoring, we propose a DSMS. Then we take the case of TelegraphCQ and briefly analyze the method of detecting duplications in it.Secondly, on the basis of existing approaches we propose a novel algorithm of detecting duplications in data stream for network security monitoring stream data. The algorithm adds a update procedure to extend and generalize the regular Bloom filter. Based on improved Bloom filter for the landmark window model, the algorithm is superior in terms of both accuracy and time efficiency when a fixed small space and an acceptable false positive rate are given.Finally, we implement the above algorithm in our network security monitoring DSMS. Our method reduces the quantity of follow-up data need to be processed and analyzed, improves efficiency and saves storage space.In summary, this paper focuses on the scenario of network security monitoring, proposes and implements a algorithm of detecting duplications in data stream, and has some practical value for stream data processing.