Design And Implementation Of Distributed Log System

In the era of data explosion, e-commerce sites and social network sites have made an unprecedented record of visits, and each consumer can produce a large amount of records. Nowadays, data is not expensive any more, digging information from massive data and using the information to improve user experience, make models for personalized recommendation is much more valuable. But it can easily meet the bottleneck in storage and computation of massive data with traditional techniques, so the distributed storage and computation technologies develop fastA distributed log system supporting open APIs is proposed in this paper. The log system not only provides a set of general data processing which includes log storage, real-time and offline analysis, but also supports custom log data structure, multi-tenancy logs, log write/query opreation authentication, etc. It allows developers to submit their application’s logs to the distributed log system, thus they can utilize the open APIs to deal with large-scale log’s storage and analysis. The system can also scale out online by increasing the node due to distributed design, and there will not be single point of failure.In this paper, investigation of current log system’s scenarios and status is firstly given. Then considering some open source log systems, the distributed log system which supports open APIs is introduced, moreover system functional requirements and non-functional requirements are described in detail. Next it comes to the paper’s main contents:system architecture, subsystem design and key technologies, followed by the system implementation, system deployment and a set of tests to verify system’s usability. Finally, a third party application’s data storage and analysis module is put forward as an example, which introduces the usage of open APIs and demonstrates the value of the distributed log system.