Design And Implementation Of Massive Log Service Platform Based On Real Time Log System

In the era of large data,if the value of the data is compared to gold,a log recording the user behavior and the system running status is undoubtedly a ore for refining the treasure.In addition,the log is also a main reference source of development,testing,operation and maintenance personnel problem investigation,and system diagnosis.However,with the development of business and continously extension of the scale of the system,various services being becoming more and more complicated,the seemingly simple task of collecting logs is becoming increasingly challenging.In the traditional enterprise application system,the log data is stored scattered by files.Such a log data has been unable to meet the needs of monitoring the real-time log and digging the full amount of the data.Therefore,it is necessary to develop a log service platform for unified real-time collection,centralized storage and analysis of the log data.The main contents of this thesis are presented as follows:(1)Combined with the actual situation of enterprises,and based on the convenience,reliability and efficiency of the system,the system architecture is defined,the system architecture is defined,the framework design of the log service platform is planned,and the system function module,the storage structure and the user interaction interface are designed;(2)Based on the Kafka message queue,the unified storage of the log is completed,the log real-time consumption and the log delivery module are realized,the storage mode is optimized,and the utilization rate of the storage resource is improved;(3)A log access service is reasonable designed,a high-throughput log transmission pipeline is provided to support a large number of real-time log to be writted in;(4)A massive log service management platform is built to facilitate the management of the log.The massive log service management platform is a one-stop service system for log data.Users can quickly complete the functions of data collection,consumption,delivery and query analysis without the development,and thus improving the operational efficiency of company operation and maintenance.