Research And Implementation Of Centralized Log Management System Based On NoSQL

With the rapid development of the Internet, log has become more and more important for the enterprise. Today, log data of enterprise is rapidly increasing every day, how to manage it become an urgent problem to be solved. In the actual production environment, due to the dispersion of different application systems’ deployment and the use of load-balancing policies, makes each application logs scattered across different servers, brings inconvenience to the log management and analysis, thus log management system came into being.Research and implementation of centralized log management system based on NoSQL is a project I implemented during my internship. At present, the project is mainly used for the centralized collection of all applications, and provide appropriate query and analysis capabilities. Query module includes real-time and historical query and can be filtered according to specific conditions. Analysis module analyzes applications and server-related performance and business metrics, such as the amount of the server run-time requests and concurrent requests, slow response and non-response interfaces.This system is implemented based on MongoDB and RabbitMQ Server. First, each application sends log which has a pre-defined data format to RabbitMQ through a PHP interface, and then stored in MongoDB, MongoDB runs corresponding analysis for the logs, and finally the results of query and analysis are presented through Web which written in PHP.Join the RabbitMQ Server as a message queue rather than directly insert log data into the MongoDB is an innovation of this system. Join the message server as a cache queue, making it easier for the system’s future expansion, and enabling the system to add a new application without the need to modify the structure. Meanwhile, the system also can preprocess the data and validate the log to the format you want. In the log analysis process, we use an improved aggregate algorithm, which implements easier and has efficient performance.At present, the log centralized management system has been used in the enterprise. This system in improving development and debugging methods, on the analysis of systems’health, achieved good results.