The Design And Implementation Of Game Log Big Data Analysis System

In face of the increasing log data,how to effectively deal with large log data has become a huge challenge for Internet companies at home and abroad,so the search of efficient and feasible large-scale data processing technology and methods have become an urgent need.Take the well-known domestic Internet company Baidu for example,its daily log data is about 700 GB,the amount of data during the holiday log data may reach the TB level,the traditional log analysis in such huge amount of data is clearly weak.At present,Hadoop and Spark are the most popular distributed computing frameworks.With the maturity of these two technologies,the use of distributed technology for log analysis has become the focus of various enterprises.In order to solve the increasing log analysis problem,a game company in Guangzhou has put forward a log processing scheme based on Hadoop,Hive and Spark,which is the design and implementation of game log data analysis system.Through the analysis of the network resources and the library resources,a large amount of relevant data and related literatures are consulted for the analysis of large data.The Hadoop framework,Hive data warehouse and real-time large data analysis and processing tools Spark and Kafka are analyzed and researched.Game log data analysis system is mainly based on SpringMVC + Hibernate + Spring to build the Web system,and select MySql5.6 as the system database.According to the way of software engineering,the requirements analysis of the system,including function requirement analysis and performance requirement analysis,is first carried out,and the system using case diagram is drawn by using Microsoft's Visio.Then the system architecture,the system database and the system data Data acquisition module,system data storage module,system data analysis module and system data display module are designed.The system display the game log analysis data through a simple and user-friendly interface,and provide the user management system,menu management and user menupermissions management for the system administrator.More importantly,the system is the first time for the R & D department to use large-scale to analyze.which will provide some experiences for large data-related development in the future.