Event Data Pattern Mining Based On Big Data

With the Rapid development of our country's telecommunications network infrastructure construction,a large number of network fault occurs,which causes a great economic loss.Recently the mature big data technology makes it possible to predict the network fault,which makes use of the existing equipment of log data and quickly locates the network fault.However,due to the large amount,high dimension and complicated structure of log data,the log is difficult to distinguish,which is a great challenge for log data mining.In order to carry out further data mining,the paper presents a new pattern extraction method for large-scale network device log.The main contents of this paper are as follows:1.We summarize the log mode extraction methods and shortcomings,as well as the main difficulties in network equipment log processing 2.We designed a method of pattern extraction for network device log,and verified the effect by the log data of 14 days of one operator in certain province.3.In order to solve the redundancy problem of pattern extraction,we used word2vec to sum and merge the log pattern,and the effect is verified by k-means and Silhouette.After a series of pattern extraction methods,hundreds of millions of network device logs are aggregated into eight thousand log modes,which effectively reduce the log data dimension and extract valuable features so that log data can be applied to data mining analysis.