Key Technology Research And System Implementation Of Distributed Log-Collection System Adapted To E-Commerce Platform

With the increase of people's consumption capacity and the rapid development of mobile internet in recent years,the way of consumption changes gradually.Online shopping is one of the most popular services of internet.The monitoring data of cluster is essential for keeping the stability of the service.Besides,user behavior on internet is an important datasource for analysis of large scale data.However,these data is collected from the log of business-system cluster.Hence,the study of designing a distributed log-collection system is of great significance.On the background of business requirements of an e-commerce platform,this paper does research on the key technology of log processing steps like acquisiton,buffering,forwarding,aggregation,cleaning and storing.Based on these key technologies,a distributed log-collection system adapted to e-commerce platform is designed and implemented in this paper.The system proposed in this paper focus on the performance of high availability,high throughput and scalability.Therefore,the reliability of log transmission and the stability of system is ensured.The design of the system has following features:To achieve the log-acquisition task on the physical nodes of business cluster,this paper makes research on the technology of multi-thread,concurrent log-collection and proposes a non-blocking solution of log collection,which is based on double buffering.On this basis,a lightweight log-acquisition component is designed and implemented in this paper.It has a high performance of log acquisition like low delay and low load.This solution solves following problems like high coupling between log-acquistion modules and business system,hardness of module's maintenance,heavy load on business system made by massive log buffer.Moreover,to support high concurrency of log writing,this paper compares ReactorModel with Proactor-Model and proposes a solution of log data forwarding,which is based on priority and master-slave Reactor pattern.In addition,a superior network communication module is designed and a data forwarding server is implemented in this paper.The server has a high performance of high concurrency,high throughput,log data transmission reliability.This solution reduces the coupling between business system and log system effectively.Besides,it solves several problems like high concurrency capacity and reliability of data forwarding on buffer layer of the system.In order to collect massive log data for further analysis,this paper does research on the framework of Flume and proposes a log system based on compound channel of Flume NG and a component of data cleaning.With the help of these modules,the system is high throughput and has large buffering space.On the premise of this capacity,the solution solves the problem of inconsistency between data writing rate and data consumption rate.In the meantime,the function of log stream cleaning during data aggregation is achieved.Hence,the log data collected and stored by the system is more in line with the needs of analysts.Time cost of data analysis is reduced afterwards.In the end,experiment of the distributed log collection system implemented in this paper has been done,including all of the modules and component in the system as well as the performance of the whole system like high availability,scalability and load balance.Throught the result of experiment and analysis,it shows that the system efficiently achieve following function like log acquisition from the business system,log buffering and forwarding,data aggregation,cleaning and storing.It also shows that the system implemented in this paper provides scalable and high available service.