A Design And Implementation Of Distributed Log And Data Collection System Based On MongoDB

In the Enterprise application systems, each system has their own log collection program.log data is stored in its own system, the format of the log also different. When users to seamlessly switch between the various systems, the log data generated by that action remain in the respective systems, so that the dispersion stored log data can not be used for the overall analysis of user behavior, even query the user login is difficult. Therefore a need to develop a system for collecting log data centralized, unified log format to achieve centralized storage for log data and analyzing.The client which distributed deployment will collect log data and transfer the data to the messaging server, on receiving the message the messaging server will write the data to the MongoDB. The cluster deployment of the messaging server can not only meet the system performance requirements, but also to ensure that the system is stable and reliable. MongoDB database using auto-sharding deployed to achieve dynamic expansion and high performance.The innovation of the system lies in the system not only collecting log data but also collecting the SQL log of the target system. The system using Druid to collect SQL logs which generated in processing user request. And according to the characteristics of both log generated in the same thread, the system using Threadlocal cleverly link the two log. Through these efforts, the system can provide the basis of the original data for the analysis of system performance.