The Design And Implementation Of Big-Data Log System Based On MongoDB

As software systems are playing an more and more important role in real life, most users are not only satisfied with the system can run, but begin to focus on the running status of the system. Exceptions, error messages are especially concerned. According to these data, we can analyze and dig out more and more valuable information. In this paper, when we metion the word "log", we are refering to the data which are recorded or generated from running systems and not related to business data. There is a wide range of log data. They are various and miscellaneous. However nowadays storage devices have large enough capacy. People can record any log data concerned by them. In contrast, it becomes a problem to find a way to deal with these data.Currently, the company’s business system generates lots of log data every day, but there is no appropriate way to handle. It takes operation and maintenance personnel a lot to maintain, but with little effects. However, if the huge amount of data is not fully utilized, it will only take up space in vain. Over time it cost more and more to maintain. Now we do have an urgent need for an automated log system, which includes collecting logs, transferring logs, maintaining logs and querying logs, and it can be an effective method to deal with big data logs. Therefore, this paper introduces the design and implementation of a new log system. Combining it with application of the big data, I implement a log system based on MongoDB. The original system which is inefficient and difficult to use changes into a more efficient and convenient system with further analyzed log data. Finally I test the system and write a sample program to prove the feasibility of the log system on solving big data problems.The project is built by maven which makes it easy to manage all stages of the life cycle. Use MongoDB to store and manage log data. You can use its high performance and built-in data processing tools for processing large amounts of logs. Some related classes are inherited to write log. And its configuration is integrated into the unified project. Use Tomcat as a lightweight and cheap server. We manage codes and documents in SVN in order to reduce the difficulty of the project management.Subsequently, the paper briefly describes the requirements of the project, as well as the design made for each demand. This section describes the improvements in project architecture, server deployment, data formats, output stream and so on, and explains why taking these solutions.This paper mainly describes the detailed design and implementation in project architecture, server deployment, data formats, output stream and so on. This section is about the work from general framework to local details. Related charts are given to help explain. This thesis is a successful attempt to use the NO-SQL to process big-data log. It will also provide an important reference for other similar projects.