Design And Implementation Of Auxiliary Collection Function Module In Network Public Opinion Analysis System

At present,there are many common online public opinion monitoring systems,but specialized public opinion monitoring systems for specific industries,fields and groups are extremely rare.Hence,When some professional public opinion monitoring departments to work in a particular area of online public opinion,they lacks of the necessary primary weapon.The purpose of this study is to make a study on the universal network public opinion.According to the practical requirement of a line of business personnel,developing a customizable,satisfying the business needs to constantly improve the auxiliary gathering module to meet the requirements of the business goals.Firstly,this thesis makes a overall systematic analysis on the problems of present network public opinion system and design functional module architecture.Secondly,it completes the detailed database design,and tested on CentOs6.5 operating system.It achieves a set of effective auxiliary gathering function system.The system mainly contains the following contents:?.According to the public opinion hot spots,as well as the professional needs of the field,the users submit the website and keywords hope to query.?.In order to solve the timeliness of massive data capture,an efficient and stable distributed crawler architecture is designed.?.Web page extraction algorithm and URL weight removal technology are used to make grasping data intelligent and visualization of analysis results.After the system is taking into efforts,it is safe and reliable,and the performance is stable.Not only to achieve all of the technical indicators,but also in the process of actual use of the user concerned about the content of the batch extraction,the basic realization of the collection efficiency of the function module of the network public opinion,the system has the alarm function.