Design And Implementation Of HDFS Monitoring And Early Warning System Based On CDH

HDFS monitoring and early warning system is a new type of cloud monitoring system.It monitors all the cloud resource in the distributed file system HDFS on the cloud platform in real-time,and makes an early warning of the HDFS status according to the obtained monitoring resource information.The cloud resource monitoring and early warning is an important direction for the current development of cloud computing.The system can provide accurate cloud resource monitoring information for enterprises,scientific research institutions,government agencies,etc.,and analyze cloud resource monitoring information to obtain early warning information of cloud resources,so as to effectively reduce management and operating costs.The monitoring and early warning system in the paper is based on the CDH platform.It performs multi-granularity monitoring in unit time and summarizes the data monitoring of nodes and clusters in real time,which help to solve the shortcomings of traditional monitoring systems,such as single monitoring indicators,slow processing speeds,and so on.It makes that the HDFS runs more stably and more safely.The main work of this thesis is as follows:(1)Design and implementation of ETL data processing: Research and compare the current cloud resource monitoring and early warning system,analyze the monitoring data processing situations,and combine the current characteristics of big data changes to design a proprietary ETL data processing process based on the CDH platform.(2)Design and implementation of monitoring model: Analyze the advantages anddisadvantages of various current monitoring systems,design and implement a new type of "active + passive" monitoring model,which effectively solves the problem of waste of resources and maintenance of connection activity in the monitoring process.(3)Design and implementation of early warning model: By analyzing and processing the monitoring data,a Bayesian early-warning model was built on this basis,and the implementation and testing of the HDFS monitoring and early-warning system based on CDH was completed according to the design ideas and models.