Design Of Massive Video Log Analysis System Based On Hadoop

In2011, with the upgrading of the domestic Internet bandwidth and the significant increase in video user penetration, the video industry gained the involvement of a large number of investors, ushered in an explosive growth.2012is a turning point in the development of video industry, how can they turnaround in the fierce competition becomes the core KPI of each video company. On this basis, it’s becoming the top priority for them to go deeper in the mining of core data.In this thesis, the existing data platform architecture of a large video enterprise is analyzed, and some problems and system bottlenecks of the old system are discovered. In order to solve these problems, on one hand, the data platform architecture of three well-known enterprises are studied, the feature and innovative part of these architecture are analyzed, on the other hand, the requirement of different teams are obtained. A log analysis system based on an open source software hadoop is designed. The system is high available, and it’s consistent of streaming log extraction, log cleaning and analysis, monitoring and job-scheduling modules. In each module, one or two open source software are used. Ultimately, the overall design concept is summarized, and some deficiencies and prospects for future system are pointed out.