Large Scale Cloud Computing Cluster Monitoring System: Design And Implementation

The cluster monitoring system observes the operation, running status and othervital information of the Node and the Cluster (physical cluster and virtual cluster),store and analyses the metrics, and displays results. It is crucial for clustermanagement and performance measurements as the monitoring data can be used todiagnose problems and to suggest remedies by both end users and systemadministrators. Scalable resource monitoring is essential to the cluster management.Monitoring System is not only used for monitoring real-time (or not-real-time)monitoring operation status, but also for load conditions remote controlling. Thus,cluster administrators can observe the entire cluster with GUI visualization to manageand maintain the cluster.Traditional Monitoring Systems always follow the way that getting the staticphysical machine information, and there are limit monitoring metrics. The Nodedaemon process has to relay on the third-part middleware, like ganglia, SuperMon,InteMon, etc. What’s more, traditional Monitoring System does not mine deeply orhas limited mining ability for persisted metrics because of its limited ability ofprediction and prevention of the Cluster. Last but not least, in terms of persistence,traditional monitoring system uses file store system or RDBS (Relational DatabaseManagement System), which is not good at storing the Unstructured and Weakrelational information.Using Data Mining and Open-Environment lightweight Node daemon developingtechnology, blending the RPC technology and NoSQL Database, this dissertation willdeeply discuss methods of developing the high efficient large-scale ClusterMonitoring System. This topic will be discussed with the following aspects:1． Research on the Open-Environment node daemon process. For the traditionalNone-Open-Environment or relaying on third-part middleware node daemon process,this dissertation will design a new node daemon process architecture which isopen-environment, and fit the different levels and environment.2． Research on the Large-Scale Cluster Health diagnosis model. Using the DataMining technology, and MapReduce method to mine and analyze the persisted metrics,and finally gives the credible evaluation and predetermination report of the Clusterhealth to the management of the Cluster.3． The design and implementation of Monitoring System. Focus on the problem mentioned above, we designed a system called Xmon. In this system, we implementthe real-time monitoring service for physical and virtual machine. Also, this systemimplements the function of alarming and health diagnosis of the cluster.System test shows that, with Xmon, we can monitor the physical and virtualmachine efficiently. Compare to the traditional monitoring system, this systemimproves the precise of the metrics and meanwhile, decrease the pressure of thecluster.