Design And Implementation Of Public Forum Monitoring System Based On Distributed Crawler

With the rise of video game,in order to guide the operation and development of games,relevant practitioners need to understand real gaming experience of the game players.What`s more,when it comes to the real working environment,game operator always face with many problems,such as the narrow range of information collected manually by the operating personnel,the low efficiency of information processing,and the subjective analysis results.Therefore,in order to meet the business needs,In this paper,a system is designed and implemented based on distributed web crawler,data analysis and data visualization to collect and analysis the game review data to solve the problems that encountered by game operators in their actual work along with other business needs.The public opinion monitoring system combines the highly efficient distributed crawler with the public opinion analysis system,which can acquire and analyze the game comment data in real time and efficiently,and display it intuitively through the visual chart.The main work of this study is as follows:A)Distributed incremental crawler of games.In order to solve the problems of large quantity,fast update and scattered distribution of game evaluation,this paper designs and implements a distributed network incremental data acquisition system based on Master-Slave architecture to collect relevant information in real time and efficiently.In the process of system design and development,a general forum information extraction algorithm is designed for a large number of games.At the same time,an efficient distributed Bloom Filter is implemented by using Redis,which greatly improves the efficiency of duplicated URL detection in the distributed environment.B)Game review data analysis.Combined with the characteristics of game review data,this paper designs and implements a game review data analysis system,which includes network neologism discovery,hot spot tracking and game emotion analysis.In view of the problem that there are many new words and improper nouns in the game review data,using left and right entropy and mutual information to realize the game specific new words discovery algorithm.In the process of practice,this paper uses the Tire Tree to transform the algorithm,which greatly improves the operation efficiency.In view of the needs of practitioners for automatic detection of hot topics in the game,the method of modifying heatindex is used to mine hot words,which has achieved good results.And in this paper,a model based on Skip Gram with extended emotion icons information is designed to implement the sentiment classification with Bi-LSTM model.C)Overall design and implementation of the system.Based on the idea of front-end and back-end separation,this study adopts the Pro backend framework,Angular + Flask,and data visualization framework,Chart.js,to design and implement a complete set of public opinion analysis system.MVC mode is adopted in the overall design,and API mode combined with asynchronous processing is adopted in task processing.The overall system has good performance,ease of use and scalability.The system constructed in this paper has many advantages,such as more adaptive scenarios,good user experience and so on.It has been applied in the game company,greatly improving the work efficiency of the relevant staff.It provides a more high-quality,convenient,objective and professional game public opinion monitoring platform for game practitioners,hoping to promote the healthy development of China's game industry.