Design And Implementation Of Automatic Deployment And Monitoring System For Big Data Platform

With the promotion of the computer application and the popularity of the smart equipment,the data volume produced through internet services are rapidly increasing.The concept of?Big data?is born at the right moment.However,people face with many problems with the rapid development of big data.For example,firstly,in the ecosystem of the big data,many softwares are open source softwares which are difficult to start the deployment and needed to exert much energy in the process from deployment to testing even for the professional personnel.Secondly,from the perspective of big data monitoring,the monitoring among the big data modules are relatively independently and the methods of obtaining monitoring are uneven.Aiming at the above problems,the paper designs and realizes a system integrating the automatic deployment and monitoring for the big data platform.The main work and the research results gained are as follows:(1)Based on the different deployment methods of big data module,pay attention to the expandability and configurability of the deployment procedure and the master-slave configuration of some modules.Design a set of automatic deployment system of big data cluster by using shell script and expect interactive language,which covers the main big data modules including Hadoop,Mysql,Kafka,Redis and Jstorm.Succeed to design and realize the automatic deployment.(2)Research and design the initialized module of monitoring information,initialize the cluster information and monitoring information to Zookeeper configuration center through automatic deployment.(3)Based on the monitoring methods of existing big data cluster,choose the system monitoring solutions centered on JMX through research and comparison,design and compile the monitoring procedure of the big data module and physical server.Choose the mongodb as the cache database module to save the monitoring information collected by the monitoring module through research on the nosql database to response to the front end of web to conduct presentation.(4)Integrate the automatic deployment module and the monitoring module to solve the problem of big data cluster and monitoring management.