Design And Implementation Of Distributed Cluster Monitoring System Based On Frequent Pattern Mining

As the basic task of data mining,frequent pattern mining has always been a research hotspot in the field of data mining.In recent years,with the rapid development of intelligent terminals,Internet and wireless sensor networks,data explosions which not only include the massive accumulation of traditional static data but also the generation of large-scale dynamic data stream have appeared in various fields of the social life.This phenomenon undoubtedly puts higher requirements and challenges on the frequent pattern mining algorithm.This thesis focuses on the frequent pattern mining algorithms,improves and optimizes the existing classical frequent pattern mining algorithms for two different data backgrounds to get a better performance of frequent pattern mining.Furthermore,taking the needs of enterprise intelligent operation and maintenance monitoring work as the starting point,merging big-data processing frameworks and frequent pattern mining algorithms,a distributed cluster monitoring system has been realized in this thesis.The main research contents of this thesis are as follows:1.Research on the frequent pattern mining algorithms for static data.Focusing on the drawback of pattern growth process in FP-Growth,which is a classical frequent pattern mining algorithm.This thesis proposes LP-Tree(linked-list Pattern Tree)structure,and on the basis of which,a frequent pattern mining algorithm for static data named LPTFPM(Linked-list Pattern Tree Frequent Pattern Mining)is proposed.LPTFPM effectively improves the efficiency of frequent pattern mining.Morever,with the Spark distributed computation framework,this thesis designs a parallel frequent pattern mining algorithm named PLPTFPM(Parallel LPTFPM).Finally,the mining performance of PLPTFPM is verified in the contrast experiments.2.Research on frequent pattern mining algorithms for data streams.With the analysis of the status-update method and mining strategy in the existing frequent pattern mining algorithm for data stream,a time-window-based data stream processing model WDPTree(Window-based Dynamic Pattern Tree)is proposed in this thesis.WDP-Tree is capable to effectively compress and dynamically update the data stream fragments.Based on WDP-Tree structure,combined with LPTFPM and Spark Streaming distributed data stream computation framework,a parallel frequent pattern mining algorithm for data stream named WDPTMS(Window-based Dynamic Pattern Tree Mining on Stream)is designed.Finally,the data compression performance of WDP-Tree and the mining performance of WDPTMS algorithm are verified in the contrast experiments.3.Design and implementation of the distributed cluster monitoring system based on frequent pattern mining.On the basis of theoretical research on frequent pattern mining algorithms and the intelligent operation and maintenance monitoring project on the information system of a grid company,this thesis analyzes and determines the functions and connections among the modules in the distributed cluster monitoring system,designs and implements a distributed cluster monitoring system integrating data collection,data storage,data analysis and results display with the big-data processing frameworks and the association rules analysis technology.Taking the dynamic monitoring on the resource of information system of the grid company and the association rules mining of the alarm data as the main work,this system realizes some core modules in the project,which provides certain solutions and technical support for the construction of the new integrated operation and maintenance system of the grid company.