The Research And Design Of Monitoring System Based On Big Data Cluster

With the rapid development of the Internet and mobile devices,an increasing number of data is generated during people's daily lives and work.When providing services to people,companies also accumulate a large amount of user data.In order to better help people make decisions and meet people's life and work needs,collecting,analyzing,and processing large amounts of data has become an inevitable option for many companies.This is especially true for the advertising industry.The needs of the advertising industry determine that it needs to analyze and process a large amount of data.In order to process big data,companies need to build big data clusters.As an open source distributed data processing framework,Hadoop has been widely used in the field of big data processing and analysis,and has become an indispensable option for companies to build big data clusters.With the rapid growth of the size of data that needs to be analyzed and processed,the number of machines in the Hadoop big data cluster will also increase rapidly.The problem that follows is that the cluster machines appear abnormal or even down,which will bring companies huge loss.Therefore,the stability and robustness of cluster operation has become a problem that cannot be ignored by companies.In order to ensure the stable and robust operation of cluster machines,it becomes particularly necessary to develop a system to monitor big data clusters.Considering the actual application,this article conducts secondary development on the basis of the open source monitoring tool Zabbix.To address Zabbix's problems in monitoring item collection,information display and alarm notification,a large data cluster monitoring system is designed and implemented.And it will be applied to practical applications.The main work included is as follows:(1)Designing and implementing a monitoring system on the basis of Zabbix to collect the performance indicators of the big data cluster,visually display various indicators on the web browser side,display the abnormal conditions of various indicators and carry out an alarm to notify relevant personnel so that they can find problems in time,and reduce losses.(2)Implementing an advertisement system,including advertisement managementmodule and advertisement retrieval module,for generating data for big data cluster analysis and processing.(3)Applying the big data cluster monitoring system implemented to the monitoring of big data clusters that process advertising data.(4)Performing functional and performance tests on the monitoring system and advertising system.