Performance Optimization Of Advertising Quality Monitoring System

With the rapid development of mobile Internet technology, websites, video, mobile clients and other channels generated so much advertising so that a lot of log data was generated when advertising. Original ad quality monitoring system encountered a bottleneck on performance and scalability with the explosive growth of advertising log data. It becomes difficult to meet business expansion. How to effectively collect, calculate and store massive log data to real-time analyze state of advertising in each area has become a new challenge.The emergence and development of a variety of big data processing technology, offer a new way for this kind of problem solving. Since the existing problems of advertising quality monitoring system on the performance and scalability, an optimization scheme was proposed from three aspects, including data collection, real-time computing and data storage. The optimization scheme uses some big data processing techniques, including the Flume log collection system, RocketMQ message queue, Storm real-time computing systems, Redis memory database and so on, to improve the performance of the whole system.In this thesis, the main contents are as follows:Firstly, the data collection layer of system is optimizing. Flume log collection system was imported as data collection layer to collect real-time data. It substitutes RocketMQ for Redis due to the limitation of the memory sizes when redis message queue is used and reconsumption ability of RocketMQ.Secondly, real-time data calculation logic optimization. Optimizing calculation logic and reducing the number of reading and writing cache IO according to characteristics that advertising system log statistics need in more than one time dimension.Thirdly, data storage hierarchical is optimizing. Data storage uses query interface service to support scaling out of Redis according to business through the configuration file. The database connection pool technology was used to improve the access efficiency of query interface service. Cache clearing service clears cache in a reasonable way of reading RDB file. It reduces the use of memory when cache clearing service workers every day. Visual monitoring of Redis monitors throughput and capacity of Redis storage system to ensure the stable operation of the storage system.The experimental results show that after the system optimization, capacity of message queue can not restricted by memory in the data collection layer, real-time data calculation reduces the number of writting cache in the data computing layer, query interface improves query efficiency and cache clearing service takes up less memory resources in the data storage layer. The performance of whole system is improved.