Design And Implementation Of Real-time Log Analysis System Based On ELK Stack

In recent years,with the rapid development of Internet application technology,the amount of data generated by the servers and all net equipments is showing explosive growth.Thus,the analysis and processing technology of massive log data is becoming a hot topic of current research in the field of big data and data mining.Most of the traditional log analysis schemes have been applied successfully,but there are a lot of problems in log collection,real-time calculation,data visualization and other aspects.This paper analyzes and compares the traditional data-process technology and the current popular big data solution,for batch processing of large data processing and real-time stream processing technology,designes and implements a real-time analysis system of data processing technology based on massive security log.There are the following aspects of the main work in the paper:1.The overall design of log analysis system.On the basis of in-depth study of ELK Stack technology and the real-time processing technology of Storm,combining the shortcomings of the traditional log analysis system,put forward that the real-time log analysis system base on ELK Stack,needs to have distributed log acquisition function,message publish/subscribe function,log data real-time computing function,distributed data storage function and data visualization function.Based on these functions,carry out the overall design of the system,analyze the requirements and goals of the system,plan and design the overall architecture,module division and business process of the system.2.Detailed design and implementation of log analysis system.Build and implement a real-time log analysis system based on ELK Stack,including distributed log collection module,message publish/subscribe module,real-time computing module and distributed data storage module.For each functional module,design the basic structure of each module,combe the business process of each module,and design and implement the basic components of each module in detail.Each function module of the system cooperates with each other,and can complete the collection,analysis,calculation and storage of the security log.3.Data visualization and system testing of log analysis system.By Nutz framework technology,I build results visualization Web services,visualizing the data in the system and providing query interfaces for log analysis results,and displaying the security situation of the equipments to users in real time.In addition,by building a test environment in the distributed cluster,all the functional modules of the system are tested.By the analysis of the test results,the functional characteristics and performance indicators of the system are verified,that the system has the characteristics of real-time,reliability,distributed computing,scalability and stability.