Design And Implementaion Of Intelligent Operation And Maintenance System For Big Data Platform

In recent years,with the development of big data technologies and applications,more and more enterprises realize that big data would play an important role in enterprises,and more and more big data platforms are adopted by enterprises.However,the big data platform,deployed on a large number of nodes and composed of many components,is not only manifold,but also complex,which makes the operation and maintenance work of the big data platform very complex,extremely dependent on professional operation and maintenance personnel.At the same time,with the development of big data application,the big data platform is more and more important to enterprises,which makes the operation and maintenance of big data platform at great risk.Intelligent operation and maintenance could complete system state evaluation,abnormal diagnosis,alarm and prediction by means of data analysis,so as to reduce the complexity and workload of system operation and maintenance.In order to simplify the operation and maintenance of big data platform,intelligent operation and maintenance of big data platform has become a trend.In this context,the intelligent operation and maintenance system for big data platform is designed and implemented.The system realizes the collection,aggregation and storage,analysis with query and visualization of the runtime data of the big data platform.By providing a series of intelligent operation and maintenance functions,the complexity and workload of operation and maintenance personnel are reduced.In this thesis,the unified scoring model of big data platform is proposed.Firstly,the benchmark is given through abnormal operation data manufacturing and artificial scoring,and then the health score of the running state of big data platform is obtained based on artificial intelligence training;the HDFS capacity prediction method is proposed to manage the storage capacity of big data platform more actively;the task scheduling method based on platform free time analysis is proposed to optimize the platform efficiency.Firstly,the research background of the intelligent operation and maintenance system of big data platform is introduced together with the researches on the related technologies;Secondly,the researches is launched from the collection of key performance indicator data,the aggregation and storage of data,the analysis of data and the query and visualization of data;Thirdly,the overall design and the design of each module of the intelligent operation and maintenance system of big data platform are specified;In the end,the implementation and tests of the system is explained.A series of functional and non-functional tests on the system verified the effectiveness of the system.