The Design And Implementation Of Log Analysis System Based On Django

With the rapid development of information technology, more and more people like the e-commerce for its convenient and fast. eBay(www.ebay.com) is one of the shopping site. Since eBay has a large number of users in the world, each time the user visits and every operation the user do will have a log to make a record, so the site will generate a lot of background system logs every day, about 1GB per day. These logs are very valuable data for the company, which can extract a wealth of useful information.However, making extraction, filtering and statistics from the logs is a very complex process, which usually requires a lot of storage space and running costs. And how to analyze the results to present to the user is also a complex issue. Therefore, in order to quickly analyze the log, obtain data and statistics presented to the user in a friendly manner, we have developed a log analysis system-Logstash for internal company use.Logstash based on B/S mode. The user obtains the desired data through the browser, it’s convenient. Logstash using the Django framework based on the MVC pattern that makes the system have a clear division of function and reduces the coupling between modules. In the storage section, Logstash uses a distributed storage, which makes a good solution to the problem of large volumes of data. The distributed storage provides a high performance, reliable and scalable function for backend storage. In the search section, Logstash uses an open source distributed search engine called Elasticsearch, enabling real-time search. In the fronthead part of the project, we use a fronthead framework called Bootstrap that enables a clear page layout, and use Ajax, jQuery and other advanced technology to show the charts to achieve a good human-computer interaction.There are many kinds of logs in eBay, each type of the logs has its own format. Therefore, Logstash log analysis system provides a scalable universal interface, user-friendly query log type and log entries to add and modify. And a certain degree of statistical data, through a friendly graphical interface presented to the company’s internal users. It’s convenient for data analysis and troubleshooting.