The Applications Of Stream Processing Technology In Network Management System

The rapid development of computer technology and network technology brought us to the Big Data Era. With the explosive growth of global data, the original standalone mode can't meet the demands of big data processing. On the other hand, many data source, such as Internet applications, business operations and scientific instruments, are producing massive and high-speed data streams every day. Mass refers to the amount of data, and high speed refers to the arrival rate of data. How to process the massive and high-speed data streams in real time, begins to attract much attention. The explosive growth of data and the need to process massive and high-speed data streams in real time, give birth to some general streaming data processing platforms and pose challenges to them at the same time. The platforms need not only support the extension of clusters but also process massive and high-speed data streams in real time.The thesis focuses on the requirements for processing alarm data in network management system, based on deep investigation and analysis of stream processing technology and typical streaming data processing platforms. When exceptions occur to agents in NMS, agents will send alarms to SNMP manager. If the network is large in scale and the number of agents is large, manager needs to filter out duplicate alarms and wavy alarms to reduce the number of alarms and help network administrators address problems quickly.The work of this thesis includes three parts:1. Investigate and analyze stream processing technology and streaming data processing platforms. Focus on some typical streaming data processing platforms, analyze its architecture, key technologies,message processing mechanism and data processing engine. Based on aforementioned general concepts and technologies, perform further analysis on some typical streaming data processing platforms, such as Storm, S4, Spark Streaming and so on.2. Analyze and point out the difficulties to process alarm data of network management system in real time. Based on the investigation of SNMP and Trap messages, analyze the requirements for processing alarm data in network management system, and point out the disadvantages of traditional processing method. Last, in view of requirements for processing alarm data in practical network management systems, give definitions of duplicate alarms and wavy alarms, and further define duplicate alarms across time window and wavy alarms across time window.3. Design duplicate alarm filtering algorithm and duplicate alarm filtering algorithm across time window based on Spark Streaming. Based on the definitions of duplicate alarms and duplicate alarms across time window, design and implement duplicate alarm filtering algorithm and duplicate alarm filtering algorithm across time window. Last, conduct experiments on the cluster and verify the feasibility of the algorithms and assess the time delay and throughput of the algorithms.4. Design wavy alarm filtering algorithm and wavy alarm filtering algorithm across time window based on Spark Streaming. Based on the definitions of wavy alarms and wavy alarms across time window, design and implement wavy alarm filtering algorithm and wavy alarm filtering algorithm across time window. Last, conduct experiments on the cluster and verify the feasibility of the algorithms and assess the time delay and throughput of the algorithms.5. Apply the algorithms to the practical network management system.In consideration of specific business needs, design and implement the receiver of SNMP message, tables in the databese and front-end pages.This thesis deeply investigates the research situations of stream processing technologies, and analyzes the architecture, key technologies and componets of general streaming data processing platforms, design and implement the alarm filtering schemes in consideration of the alarm stream's features and requirements for stream processing. So the research work and results of the thesis provide novel ideas to the design and application of network management system, and has important guiding significance and reference value for the applications of streaming data processing platforms in traditional scenarios.