Design And Implementation Of The Platform Of Operation And Maintenance Log Collection And Analysis Based On Hadoop

With the development of cloud computing and big data technology, more and more data centers have been set up by the government and enterprises to run a variety of business systems. Business system is rapidly increasing, the number of nodes reaches tens of thousands or even more, and the operation and maintenance log size increases rapidly, and the lack of effective analysis framework. Log analysis has always been an important way to reflect the operating status of the application system and user behavior, especially the development of Internet technology, the analysis of the log to deal with a lot of innovative direction. However, the hardware of the system is provided by many manufacturers, the software is developed by a number of manufacturers, the log format and the various forms of the log, the lack of a unified access mechanism. A large number of repeated alarm logs, the same fault caused by the system between the log duplication, the lack of effective log clustering method. And the system generated by the operation and maintenance of the data is not related to the analysis and depth of the use of the data, operation and maintenance of low intelligence.To study the multi-source heterogeneous logs loading technology, support a variety of formats, various forms of log data distributed uniform load, solve include relational database data, text data files, acquisition scripts that generate data and user-defined business data, such as multi-source heterogeneous data access problem is one of the main contents of the thesis.Distributed computing framework and storage technology, to provide the basis for the operation of the upper layer of distributed computing and storage capabilities, to provide a standardized interface, support for the continuous construction of operation and maintenance analysis. To provide a unified data storage and data management interface, the operation and maintenance log offline clustering and online real-time classification of the ability to deal with.Research operation and maintenance log association analysis technology, the log, the system indicators of the association analysis and integration, mining association rules, to build the knowledge base of association rules. Predict the system operation status and fault in advance, and find the system fault in time.This paper obtains the main achievement is based on the Hadoop build a distributed log acquisition and analysis platform, the platform provides support for operation and maintenance of large data analysis based support frame, support online and offline distributed data processing capabilities, while providing a unified data access and storage interface; data loading subsystem is designed and implemented, the system for the collection of multi-source heterogeneous operation and maintenance logs provides effective different data acquisition scheme; design and implement the operation and maintenance of log association analysis subsystem provides:feature extraction, clustering, classification and association analysis log of offline and online data analysis ability.