The Research And Application Of Hadoop Cluster Monitoring System

With the rapid development of information technology, the traditionalinfrastructure are constantly migrating to the cloud platform, and cloud computing arebecoming a mainstream in2013the government information construction in thecoming years.Which are built on Hadoop cloud computing platform is the mostmainstream of popular applications, is the world’s major IT giants widespread use anddevelopment.With Hadoop cloud computing platform to build a successful andcontinue to use the process of expansion, Hadoop its own monitoring system has beenunable to fully competent to understand and master the cluster tasks, it also bringschallenges for cloud management and monitoring platform.At this time a full-featured,allowing managers to grasp at any cloud platform information in a variety ofmonitoring systems is particularly important.The current, resource monitoring studiesfocused on grid computing, there is little talk of the Hadoop platform monitoring, andmonitoring of grid computing does not state a fast response Hadoop platform, farbelow the Hadoop platform monitoring requirements.This paper first carries on the analysis and comparison of some of themainstream of the existing monitoring system, and with the Jiangxi Telecom’sHadoop cloud computing platform as the background, in view of the networkstructure,the requirements of information security platform, business requirements, functionalrequirements and other aspects, through the analysis and planning, the monitoringtool Nagios and Ganglia these two sets of open source.According to the advantagesand disadvantages of these two sets of software, complement each other, and inaccordance with the actual situation existing Hadoop cloud platform to develop anddesign a monitoring system capable of complete and real-time monitoring Hadoopplatform.The monitoring system can monitor Hadoop platform application tasks, eachnode in the cluster performance monitoring, information security considerations for adistributed architecture, a variety of monitoring indicators and information can beintuitive way to display the Web page, administrators more easily fully understand thecluster management and cluster information.Monitoring systems in the cluster can fail in many ways rapid alert notification management, improve the operating speedcluster monitoring system for an existing Hadoop can complete storage monitoringinformation to achieve a Hadoop HBase for monitoring data storage and querypresented evidence for health diagnosis and future expansion of the cluster, the finalfunctional testing and performance testing related systems.