Design And Implementation Of Real Time Log Collecting And Processing System For Televison Users

In the Internet era,people's work and life rely on the network and information technology to provide services.Users of these services continuously generate log data.These log data are valuable.How to use these data effectively is currently a common problem faced by Internet service providers.With the rapid development of Internet technology,traditional radio and television services have also shift to Internet service providers.Therefore,how to use log data to improve service quality based on the analysis results of log data is an important way for radio and television service providers to improve their market competitiveness.Relying on the wired network TV business of a TV ISP,this article designs and implements a log data collection and processing system for TV users,based on the disadvantage of existed services provided by the company.The primary difficulty of this subject is complex network topology,large number of concurrencies,and large scale of network traffic.Different from the common server-side log collection,the TV user logs of this topic need to be posted to the collection system by HTTP requests.The report request will cause numerous pressures on the collection system.Therefore,this paper mainly does the following research work to solve this problem:Data collection module.A distributed collection server cluster is designed based on load balancing through Nginx and distributed technology.Log Data are collected by Tomcat server cluster.Data cache module.Based on the idea of multi-level cache,an asynchronous and non-blocking I/O model was designed with using Redis and Kafka message queue.Data processing module.Based on sliding window,similarity calculation,regex matching and other methods,log classification,log cleaning and anomaly detection are achieved.Data storage module.This module mainly uses MongoDB,which is a non-relational and document database,to achieve efficient storage.At the same time,a visual management module and distributed system monitoring module used by the system administrator are implemented,which saves many manual operations of Site Reliability Engineers.Finally,the basic function test of the collecting system implemented in this paper is achieved along with the performance comparison test of the existed log collecting methods.The results show that the scheme implemented in this paper improves the system performance compare to the existed scheme in service.