The Design And Implementation Of Data Real-time Collection And Processing Subsystem In Network Monitoring System

In recent years,the broadband Internet presents a rapid development situation in China,the operators continue to expand their network scale.It becomes more and more difficult to monitor and manage the network equipments and network services fast and conveniently.The network elements monitored by the network monitoring system are generating and reporting data all the time,but these data are the original data,which need the system to analyze and process to extract useful and normalized data information.In the face of large network equipments and various network services,it is a difficult point for the network monitoring system to collect and process the indicator data easily and efficiently.The data real-time collection and processing subsystem introduced in this thesis is used to collect and process the indicator data in the network monitoring system conveniently and efficiently.The data real-time collection and processing system is required to be able to collect common indicator data,deal with the service data reported by file,UDP and database interface,and at last represent the result in the form of report.During the data collection and processing,it is necessary to solve the integrity and uniqueness of the data.The real-time and scalability of the system are also the focus of attention.This thesis begins with the data processing technologies,introduces the main features and application scenarios of these technologies,then analyzes the characteristics of indicator data collected and processed by the network monitoring system and introduces the design idea of the system.Next,the thesis briefly introduces the related technologies used in the system.After that,the requirement analysis and the overall design of the data real-time collection and processing system are carried out,and the function scope and the overall structure of the system are given.This thesis focuses on the detailed design and implementation of data real-time collection and processing subsystem in network monitoring system,mainly introduces the detailed design and implementation of data collection subsystem,data preprocessing subsystem and report subsystem,at last,the function and performance test results of the system are given.In the last part of the thesis,the author summarizes the work,and looks forward to the future of the data real-time collection and processing system.