Design And Implementation Of Network Security Log Management Analysis System Based On ELK

In recent years, the wave of the Internet is giving people a huge change in the production and life, on the other hand, cyberspace security issues have begun to highlight. Tens of thousands of devices generate a considerable number of security logs per day, and the management of network security logs is a very important defense in the security arena.Through the network security log management and analysis, we can improve work efficiency, and it is helpful to solve problems in the business.Now many network security log analysis system have been put into using. With the development of technology and business, it should improve the system capacity.In this paper, log management analysis system is our main research object based on the practical problems of network security log management and analysis system. Combining the problems of log system and the research status of log analysis algorithm, we design the management analysis system of network security log based on ELK for the management of large data log and analysis work. We enhance the system crawl data and analysis of alarm functions, especially log retrieval capabilities, and large data storage technology combining to big data technology. It enhances the system performance.The main contents of this paper include the following aspects:1. In-depth study of the log system on the key technology and log analysis algorithm for the future development trends, the main technology and open source tools are reviewed.2. Design the network security log management analysis system based on the ELK. The system is divided into six modules. It completes the log collection function, log processing function, log retrieval function, log display function, log storage function and log analysis and alarm function.On the basis of the overall function design,it carries on the concrete demand analysis and the design to each function module.3. The implementation of the designed log system is mainly based on the ELK technology stack, combined with the Scrapy framework, Redis Message Queue and HDFS storage technology, the six modules were implemented. And PCA and one-class SVM algorithm are the main analysis algorithm of the analysis module, it completes the important function of log analysis.4. The results show that the new log collection module and the log analysis-alarm module can improve the system capacity. The system has certain advantages in dealing with massive log.