Design And Implementation Of Operation And Maintenance Management System Based On Hadoop Cluster

In recent years,big data acquisition,storage,management,processing,analysis and other related technologies have made significant progress.There are many successful big data application technologies,but in terms of its effect and application depth,the current big data application technology is still in the primary stage of development,and the technical system is not perfect.Among them,one of the difficulties is to achieve efficient and reliable operation and maintenance guarantee for the large capacity server cluster.With the development of operation and maintenance technology,the data scenarios and tool scenarios are changing with the times.The operation and maintenance scenarios are more complex,and the data processing and maintenance are more and more huge.From the lack of data at first,to a moderate amount of data,and then to the need to process and maintain a large number of data.These factors bring many problems,such as data isolation,unable to fit the business,complex tools,unable to effectively integrate data,unable to extract data value,etc.Data is always the relationship of multi-dimensional management,so it is difficult to obtain effective information management and information analysis capabilities if it is unable to quickly extract the core concerns of enterprises.There are data,tools and teams,but they lack the ability of centralized alarm,unified display and correlation analysis,which is a common problem faced by many enterprise cluster operation and maintenance.Considering this situation,this thesis designs a Hadoop cluster operation and maintenance management system based on Hadoop cluster,which is the mainstream system platform of big data processing.On the one hand,it realizes unified alarm,centralized log output and audit of user operation behavior,which is convenient for users to control the cluster;On the other hand,it integrates data,refines data value,provides load analysis,key component diagnostic analysis and data visualization services for clusters,optimizes the allocation and use of cluster resources by enterprise users,and provides strong guarantee for the supervision and maintenance of clusters.In order to better realize the design and research of Hadoop cluster operation and maintenance management system,this thesis first studies the related concepts,theoretical knowledge and application technology of cluster system.This thesis investigates the research and development of cluster operation and maintenance management technology at home and abroad,as well as the common problems and market demand in the production practice of enterprises,and summarizes the core concerns in the process of cluster operation and maintenance management.On this basis,combined with the application software system design theory and technical methods,the requirements of the cluster operation and maintenance management system are analyzed,and the results are elaborated by using the detailed function model and data model.The design objectives and infrastructure of the system are defined,and the detailed function module design and implementation of the target system are finally guided.The main work of this thesis is as follows: according to the requirements analysis,the system function module is divided,each component and function module is designed and implemented in detail,then the system code development and part of the function script development are carried out,finally the system deployment and function test case are completed.Considering the reusability of components and the readability of architecture,the system architecture follows the MVC hierarchical structure of software design paradigm,and is researched and developed based on spring MVC framework technology.After testing,the target system designed in this thesis achieves the expected requirements,reduces the time cost and learning cost of enterprises and users to use and maintain the cluster system,and has a good application prospect.