Design And Implementation Of Data Processing Subsystem In Application Performance Management System

With the advent of the era of big data, putting forward a scientific and rational method for application performance management (APM) are more and more drawing attention of enterprises. On the one hand, through APM, enterprises can discover and solve problems in the system, and even predict when the system will break down; on the other hand, APM can provide performance optimization solutions to save the cost.Performance analysis of APM system is based on the collection and pre-processing of performance indicators to ensure data real-time,availability, accuracy. The diversity of indicator data, the complexity of distributed clusters, the explosive growth of data volume makes the data processing of APM system face great challenges.In order to deal with complex and diverse data in APM system quickly and accurately,and provide support for performance analysis, this paper designs and implements a distributed data processing subsystem.This system deals with the massive data of a large number of monitored tasks in the distributed cluster, can shield the heterogeneity of the underlying system, effectively collect all the data that scattered on different machines and process data type conversion, dirty data cleaning,smoothing, filtering and other real-time processing. The system is constructed based on streaming computing platform Storm, and uses the message middleware Kafka to receive and store original data, designs and implements the cleaning method with the combination of the median and EM algorithm based on parameter type, and abnormal data detection and filtering method based on least squares method, which greatly improve the quality of original data, provides a good data foundation for application performance analysis. At the same time, the system also has good scalability, availability and performance.This paper takes APM system as the background, introduces the design and implementation of data processing subsystem in detail. Firstly,the research status of APM system in China and abroad is introduced,with the main contents of this paper according to the data characteristics of target system. Then, the frameworks of distributed data processing such as Storm and Kafka used in the implementation and the background of data processing method such as data cleaning and data quality enhancement are explained. After then, gives the introduction of project background and the performance evaluation method, explains the system goal from both functional and non-functional requirements. Then this paper introduces the overall design of the system, and uses the interactive interface and interaction flow chart to illustrate the relation of modules.The implementation details of the system are described with the class diagrams and flow charts. Then, design the test cases to verify whether the system is consistent with the requirement. At last, the main work of this system is summarized and the future work is forecasted.