Research On Fault Early Warning Based On Big Data Mining

As human society entered the information era,the concepts of big data are slowly filtering into people's minds.More and more enterprises are aware of the importance of data resources which they have.Efficient information analysis ability and the transformation ability of data value have become the core competitiveness of enterprises in the complex and changeable market.For operators,they have vast amounts of data,and mining the value of these data can help them gain advantages in the fierce market competition.Therefore,how to establish a set of scientific analysis methods,also effectively use big data for efficient system operation and maintenance is a very important research topic.In the context of efficient operation and maintenance of operators,this paper constructs anomaly detection algorithms based on network elements' log data to support the requirements of fault early warning.In addition to anomaly detection algorithms,a log-based fault warning system is designed and implemented to complete large scope and complicated computing tasks.Taking into account the analysis requirements of operation and maintenance personnel to analysis specific logs,this paper use periodic sequence characteristics and continuous time influence factor to solve the problem of time series analysis,also propose an abnormal detection method based on modeled log's time series.On the other hand,in order to cope with the global anomaly detection demands,so operation staffs are easy to understand the macroscopic state of network elements' log data.The feature reduction algorithm is applied to the processing of high dimensional log data,large-scale flow data processing technology is used in this paper to improve density-based outliers detecting algorithms' accuracy and efficiency.In this paper,the real data in the actual scene of operators is analyzed,the comparison and verification of the analysis results and the actual fault cases fully demonstrate the effectiveness and accuracy of the algorithm.Based on the algorithm,this paper constructs the concrete realization of fault early warning system based on big data mining,visualization method and anomaly detection algorithms are integrated through software platform,which enhance the interaction of the analysis system.Considering of business requirements in large scale and real time data analysis scenarios,a distributed processing technology based on flow data is studied and applied to establish real-time fault warning scheme based on log data.The scheme can accomplish the collection,storage and analysis task of large scale logs in production environment,also can discover the anomalies of network devices and make early warning according to algorithm models and dynamic rules.