Design And Implementation Of A Acquisition, Processing And Storage Data System Oriented To Cluster Monitoring

With the rapid development of today’s computer technology and the growing popularity of intelligent terminals, everyone can use the Internet services anytime anywhere. The concept of the "Internet +" has become common in daily life. Therefore the stability of the service will be particularly important in cluster monitoring. Monitoring data is essential for detecting the status of the cluster service. Faced with the problems of the existing monitoring systems such as the acquiring data are incomplete, the methods of data collection are not unified and the monitoring data are not reliable enough, the study of designing a data system of acquisition, processing and storage for cluster monitoring is of great significance.On the background of the operational requirements and framework of an Internet product, this paper designed a data system of acquisition, processing and storage suitable for cluster monitoring. The system consist of 3 layers which are collecting layer, processing layer and storage layer. The core functions of the system include collecting monitoring data, processing monitoring data as well as stroring monitoring data.To provide comprehensive, timely, complete monitoring data and build a highly available system, the design of the system has following features:1, This paper proposed a simple and efficient method for the management of monitoring data according to the uncomprehensive, inefficient acquisition and unified of the monitoring data. This paper designed a storage structure based on matrix structure according to the method mentioned above, which is used on monitoring data acquisition and statistics, effectively reducing the cost of the statistical data, unified collecting methods to improve the reporting efficiency.2, This paper proposed a high-performance storage architecture base on My SQL according to the performance deficiency of monitoring data storage.This paper customized table structure for monitoring data storage as well as combining with partitions technology to achieve efficient data storage layer, which compressed the monitoring data improving the efficiency of stroage as well as the writing and reading data.3, This paper designed a high availability system architecture according to the insufficient capacity of the system disaster recovery of existing monitoring system, which ensuring the integrity of the data. With high availability architecture this papaer also designed an effective disaster recovery strategy for data transfer between layers of the system to ensure that system remained stable service in the failure.By testing and analyzing the system, this system achieved efficient monitoring data collection and reporting, high-performance for mass data storage and reliable continuous service.