Design And Implementation Of A Component-based Automated Deplo Yment And Management System Based On The Big Data Platform

With the progress of human science and technology and the update and iteration of information technology,big data has become an opportunity for the development of information technology in all sectors of society.However,the process of big data components from installation to deployment to use is very complex,so practitioners have a strong need for a fully functional and highly integrated big data platform system.However,the development of a fully functional and user-friendly big data platform system also faces many problems.For example,in the whole big data ecosystem,the components are so complex and diverse that even professionals need to spend a lot of energy from deployment to test completion.Second,when there is a technical problem with the big data-related components,developers often need to look at the log to solve the problem.But there are so many big data components that it is difficult to find log files.Thirdly,the big data components often need real-time monitoring and warning and notification system to avoid large-scale component failure caused by untimely handling.Fourth,in order to ensure load balancing,it is often necessary to manage the internal resources of the component.Fifth,some big data platforms in the market,such as Ambari,lack the internal management of big data component resources,the management of logs,the interface is not very friendly,and the lack of monitoring and management support for non-big data componentsAiming at these problems,this paper designs and implements a set of big data platform automation deployment and monitoring management,log alarm and resource management integration system.It specifically involves five modules of big data platform:automatic deployment module,monitoring and warning module,log management module,service management module and resource management module.And has realized the component one-key deployment,the component abnormal monitoring real-time alarm,the component log management,the component setting management,the component resource visualization management five functions respectively.System development tool,using data one-stop full link through a visual interface easy to learn easy to operate and make up the existing shortcomings of big data platform,realize the big data platform of enterprise fleetly,application deployment and operational monitoring,greatly reduce the application development and operational costs,big data for big data developers mass multi-source heterogeneous data collection,storage,processing,access,and other one-stop large data capacity.