Hadoop Cluster Monitoring System Based On Ganglia

With the rapid development of cloud computing,distributed cluster system has been widely used,the development of cloud computing technology has become one of the important economic and technology development strategies of our country,researching and developing the cloud computing and transferring the original information and data to the cloud computing platform will be the mainstream direction of the informatization construction of the government in the next ten years.Hadoop as a data analysis and data mining software,has become an integral part of the cloud computing platform.Cloud computing provides more and more services,the amount and types of data which needs to be processed are increasing,at the same time,the rapidity and accuracy of the data processing results are also increasing,so it is very necessary to monitor the cloud platform.The increase of the data amount makes the monitoring and control system of Hadoop can't monitor the resources effectively,the existing monitoring software can only be used to monitor the applications and services of different platforms and systems and can't monitor all the resources of cloud platform.In order to solve this problem,this paper,by using the Ganglia software,designed and developed the Hadoop cluster monitoring system.The main research work is as follows:1)Realized the monitoring of Hadoop cluster system.After analyzing the advantages and disadvantages of the existing cluster monitoring software,the method of Ganglia combined with Nagios is chosen to monitor the cluster system.Finally,by using the drawing function of RRDTool,the monitoring information was shown to the management personnel through the Web interface in a chart form.2)The warning service realized by Nagios.Through the configuration of the monitoring period,monitoring contacts in Nagios,host and host group,service and service group,and other information,the cluster monitoring system can make managers receive the warning message through the interface,email or SMS,even if they are not in front of the monitoring system.3)Realized the persistent storage of data.RRDTool database is a ring database,storage capacity is limited,in order to avoid the disadvantages of the unabiding storage of data,the Java API interface of HBase was used to transfer the files in the RRDTool database to the HBase.4)Verified the control effect of the cluster.After the completion of the deployment and the implementation of the monitoring system,the effect of monitoring required in the demand analysis is verified,such as the monitoring of the data blocks,the monitoring of node task,email alert service and SMS alert service,the monitoring system is confirmed to meet the monitoring requirements.