The Design And Implementation Of A Log Analysis System Based On Distributed Computing Platform

With the rapid development of the Internet, log the amount of data generated bythe network every day is huge. How to solve the problem of massive log dataprocessing has been the field of log analysis is a very important research topic. With therapid development of network technology, data on the Web is the exponential form ofrapid growth, and the data on the Web has a massive, diverse, heterogeneous, dynamicchange, which makes centralized log analysis based on a single node the platform cannot meet the massive data network analysis requirements. To design a common scalableplatform to effectively deal with the massive log data, and analysis of Web page visits,the inevitable choice of the Internet enterprise development.For the problem, the analysis of the key technical basis of the existing distributedstorage and computing, combined with the analysis and research on the Hadoopplatform was designed and implemented based on the mass of the distributed computingplatform log analysis system, and use this system for Web visits statistics. This paper,the various functional modules of the system described in detail and presented in thispaper distributed platforms, the efficiency of experimental analysis. Experiments showthat the analysis system, through multiple resources to complete the original workundertaken by a node, whether it is in the implementation of data processing or tasks, itsefficiency is higher than stand-alone centralized environment-based Web log analysis.