| Log contains a lot of information,which is an important basis for detecting anomalies,with the dramatic increase in the amount of log data and the gradual enrichment of content,traditional manual log analysis is inefficient and requires some experience,which can not meet real-time and automatic log abnormality detection?This article analyzes the log of China Unicom's SIM card management platform,the log data of the platform is large and rich in content.However,the lack of effective log processing and analysis means that it is impossible to detect and stop the abnormality in time..Based on the above,the main work of the dissertation includes the following three aspects:one is to investigate the principle and mechanism of Hadoop and Spark,which are big data processing technology,and research the real-time log collection,distributed message queuing and streaming processing technology.The second is to analyze the requirements of SIM log analysis system.With the goal of automatic,real-time and rapid analysis,a distributed SIM log analysis platform is designed and implemented on the basis of Hadoop cluster.Solving the problem of lack of memory in log processing through file merging,pre partition,and adding random prefix.Then through the log collection,message queue,streaming technology to achieve real-time log processing.Finally,the visualization of the results is realized.The three is to analyze the access logs and system log logs respectively,and use a variety of anomaly detection algorithms to detect outliers and compare the effectiveness of different algorithms to achieve automatic real-time anomaly detection.The final realization of the SIM card platform log analysis system has achieved automatic,real-time,rapid processing of logs and detect anomalies,which is an application of intelligent operation and maintenance,which significantly reduces system operation and maintenance labor and time costs,and improves the accuracy of anomaly detection. |
PDF Full Text Request