Font Size: a A A

Key Technologies Research On Network Security Monitoring Data Streams

Posted on:2009-06-17Degree:DoctorType:Dissertation
Country:ChinaCandidate:L TianFull Text:PDF
GTID:1118360278457121Subject:Computer Science and Technology
Abstract/Summary:PDF Full Text Request
With the rapid development of computer-related technology and increasingly extensive network applications, the Internet plays an increasingly important role in our lives. On the other hand, various of security events become the obstacle of Internet's development. Network security monitoring is important both on network maintenance and infrastructure & information system security. One of the most important challenges faced in network monitoring and data processing areas is how to transmit and process the distributed massive monitoring data in an efficient manner, thus to provide support for various follow-up applications.Based on the background of network security monitoring, this paper focuses on the above challenge, and researches the cost-efficient solutions for several basic problems on distributed network monitoring. The main contributions are concluded as follows:We firstsly address the problem of continuous extreme queries (MAX or MEN) over distributed sliding window streams, and develope an effective pruning technique to minimize the number of elements to be kept. A communication-efficient method called DTA is designed, where remote nodes delay the data transmission as late as possible, and adopt the pruning strategy to filter local stream tuples, which is quite efficient in communication reduction. Further more, an efficient algorithm called MCEQP is also proposed in the coordinator, where multiple queries are processed in a correlative manner and some computation and storage resources are shared by them, which is more efficient than processing them separately. Theoretical analysis and experimental evidences show that about 99% data can be discarded by the pruning technique, and only O(logN) key points need to be stored for exact answer of extreme query if the data is independent. Comparing with the method transmitting each data to the coordinator, only 10% communication cost is needed by DTA. The MCEQP method is also proved to be more efficient than other existing ones.Secondly, a method called CEM is proposed to reduce the communication cost for continuous threshold monitoring, which utilizes the relationship among objects and processes them as a whole, therefore achieves better performance than those who holding each object separately. In specific, the object with largest value is chosen as the representative, and adjustment factors are used to guarantee that local value of representative object is also the largest one in each remote node. Only the representative object needs to be monitored continuously as long as all the local constrains are valid. Experimental evaluation shows that about 70% communication overhead can be saved by CEM.Thirdly, we notice that the skyline computation in network monitoring environment is quite different from the existing ones. The problem of continuous skyline computation on streams with random additions and deletions is raised, and two solutions called BCSC and GICSC are presented. BCSC is a tweak method which adapts the existing algorithms to the specific scene, while GICSC adopts a novel grid-indexed data structure and smart initialization and maintenance methods without having to process all the data points, therefore achieves low running time and about 10 to 10 data tuples can be processed per second. Since there's no assumption limitation of stream characters, the BCSC and GICSC algorithms are more adaptive.Fourthly, a generic method based on prediction models is researched for communication reduction. A detailed framework is presented. Three particular prediction models are improved and compared with existing ones. Analytical and experimental evidences show that the proposed approach performs better on both overall communication cost reduction and prediction query processing.Finally, based on the above works, a data stream processing engine prototype system named StarAnalysis+ is designed and implemented. The project is supported by the National High-Tech Research and Development Plan of China ("863" Plan), and is used in the "Beijing Olympic Network Security Analysis and Display System" for validation.In summary, this paper addresses the problem of cost-efficient processing for network security monitoring applications, and proposes several specific solutions for three basic queries and a generic framework for communication reduction. This is a promotion of large-scale network monitoring data processing on both theoretical study and practical applications.
Keywords/Search Tags:network security monitoring, distributed data stream, cost efficient processing, extreme query, threshold monitoring, skyline query, communication cost, prediction model
PDF Full Text Request
Related items