Log Prediction And Faults Diagnosis System Based On Sequential Pattern Mining

Hadoop is the most popular platform in storage and computing of Big Data,with the features of cheap,stable and easy to scale.Though it is widely used,it's failures detection still depends on human experience.In fact the key information all conceals in the logs,but it's a hard work to detect it considering the diversity of experience caused by system differences.We design and complete a system for Hadoop log prediction and failures detection,trying to resolve the problems.This paper is all about how to design and complete a log prediction and failures detection,and abstract log data as time series data.First we designed a log library helping developers to output logs.Then we preprocessed the data using MapReduce,creating index structures from the data.We used frequent sequence patterns mining with time constraints and bit scoring with penalty to create the predictive rules,and based on these rules we set up the log predictioner.We also used index structure to optimize the mining algorithm.Motifs with features were used to classify the failure log.We proposed a novel classification formula based on motifs matching and a motif mining method.We proposed a novel pattern matching method based on dynamic programming because pattern matching was used a lot in our system.Finally we transplant frequent sequence pattern mining,rules generating and motif mining to MapReduce framework so that we can deal with big data.