Design And Implementation Of Data Processing System For CDN Log Service

With the continuous development of science and technology,more and more Internet companies in the business process of expanding,in order to improve customer service quality,choose to visit the customer's access data to the CDN(Content Delivery Network)node.However,it is the main problem that the data processing system needs to solve the continuous improvement of the quality of the online node service in the face of the increasing number of CDN cache nodes and the monitoring log and equipment monitoring log.Based on the characteristics of CDN log service,the thesis makes the corresponding system design and implementation,and applies it to the actual business.The primary works in this thesis are presented as follows:(1)Based on the scalable,reusable and stable design requirements,the overall framework design of the data processing system for CDN log service is planned.Based on the full investigation,the functional modules and technical schemes of the system are designed;(2)Through the CDN cache node on the deployment of the corresponding timing program to achieve the log collection,program monitoring,log backup function,the log in the transmission process,through the load balanced processing to the local cache system to achieve the data Multiplexing function;(3)Using Spark log analysis technology to realize real-time calculation and off-line calculation of log flow,and save the results of analysis into distributed file system and high-performance storage system to meet front-end business logic processing or other business needs;(4)To carry out the corresponding log archiving process to meet the specific needs of customers to enhance the integrity and availability of the system.This thesis designs and implements the data processing system for CDN log service that has been used online.The system realizes the real-time acquisition,transmission,storage and analysis of the user access log,billing log and device monitoring log of the online CDN cache node,and outputs the information such as real-time bandwidth and traffic distribution to the online device.Abnormal traffic for real-time monitoring and alarm,on the one hand to help customers real-time view the current situation of the specific services,on the other hand for the operation and maintenance personnel to provide the actual situation of the alarm equipment.The system has been stable since its on-line,consistent with the intended design goals.