Research And Implementation Of Cluster Monitoring Data Collection And Analysis System In Cloud Environment

Datacenter that based on cloud computing has been deployed by more and more enterprises. In cloud computing datacenter, resources are so highly centralized that enterprises are facing a problem that how to effectively monitor all kind of resources and improve the quality of service when building their private cloud system. Some public cloud vendors provide monitoring system in their cloud, but private cloud system such as OpenStack has no perfect monitoring system. To meet the demand of enterprise's cloud environment monitoring, a cluster monitoring data collection and analysis system of cloud environment designed and implemented in this paper.This system could collect basic information and performance monitoring data of virtual or physical machine and support user to add custom collection items, which solved the problem that is hard to meet the diversity of monitoring data in private cloud environment. Content in this paper mainly has three key points. Firstly, the system uses target node automatically registered and heartbeat mechanism to synchronize node status to achieve automatic discovery and status monitoring. The system also provides a Web-based control view which can be used to check the detail information of each node. Secondly, the system provides monitoring data RESTful API interface that allows third-party components such as data rendering module to use, which make modules independent and scalable. Thirdly, the system provides a performance analysis module, regular tasks are submitted to the performance analysis platform everyday to compute performance and cluster health. Using the results of performance analysis, clusters have the ability of DRS(Dynamic Resource Scheduler) and provide a reference to the operation and maintenance trouble shooting basis which improve the overall cluster utilization.The system has several modules which have APIs to interact with others. This design method can simplify the complexity of the system. Final testing shows that the design of the system is reasonable. Functions basically meet the expected demands. In company's application, it can support to tens of thousands of target machines. Performance analysis module gives accurate results and the utilization of cluster was improved to 99.98%. In short, it's a feasible data monitoring and analysis system.