Hadoop Platform Monitoring And Optimization

With of rapid development of computer and mobile Internet technologies,we have entered an era of explosive growth in data volume.Industries are constantly generating large numbers of complicated data resources,such as social network,E-commerce transaction,Internet financial,and biological health.People hope to be able to obtain useful information from these large amounts of data resources that are closely related to user behaviors,thus improving people's lifestyles and quality of life.Under such strong market demand,Hadoop has attracted more and more attention from scholars.As an advanced big data processing tool,it has been gradually applied to various fields to help enterprises process big data.However,most platform nodes are cheap machines.With the growth of the platform size,how to efficiently manage and maintain the platform and ensure that the platform works stably and efficiently has become a big problem.Therefore,the monitoring and optimization of Hadoop has gradually become one of the hot topics of many scholars and users.This article first introduces the basic concepts of Hadoop,gives an overview of the current main distributed monitoring systems and monitoring technologies,and details the functional requirements,architectural design,and key technology cores of distributed computing platform monitoring systems.Then it analyzes the method of optimizing the Hadoop platform from different angles,and introduces the optimization method of Hadoop configuration parameters in detail.The main contents are as follows.For the problem of not meeting the monitoring demand,this article develop a fullfeatured Hadoop monitoring system by using other monitoring tools to improve data collection,display and alarm functions based on the Nagios monitoring tool.The system excavates data from the Hadoop JMX interface and system resource files,and provides a flexible,multi-view visual interface.The monitoring system can also send alarm information by mail,SMS,QQ and We Chat.At the same time,it also provides remote access interfaces for viewing monitoring information.For the problem of how to select Hadoop configuration parameters,this article introduces the method of using SVM and genetic algorithm to select the best configuration parameters,and verifies the feasibility of the method by experiments.Compared with the default configuration parameters,this optimization method can improve the implementation efficiency of the Hadoop platform by 19.1%.