Design And Implementation Of The Collection And Analysis System For Microblog Public Opinion

With the rapid development of the Internet,microblog has become the main way to share network information and it brings much convenience to people's lives.However,due to the lack of effective monitoring and management of information,the society has brought false information and rumor which may undermine the good network of public opinion and endanger public life.Therefore,we need a collection and analysis system for microblog public opinion to real-time and effective monitor microblog public opinion.It is able to detect hot spots and public opinion on the subject of emotional intensity computing,and for negative news appropriate to guide public opinion.In this thesis,the system of collecting public opinion microblogging technology and public opinion analysis techniques both in-depth research and analysis,and the system was designed and implemented,the major work are as follows:(1)On the basis of the Scrapy crawler,this thesis designed an efficient distributed crawler system based on ZeroMq.The thesis proposed the method of combining bitmap with Hash index to solve the massive conflicts of URLs.Because the microblog made anti-reptile restrictions for network crawler,so this thesis proposed the agent node collection and regular collection combination of strategies to improve the efficiency of data crawling.In order to increase the amount of data,this thesis also incorporates microblog providing the opening platform of service interface to crawl microblog data and expanded the amount of data.(2)This thesis proposed an improved method for topic microblog emotional intensity calculation.Firstly,it builds an emotional dictionary,then takes a pre-process to the subject of comments;it calculates its emotional intensity based on microblog's comments emotions at the same time;finally optimize the emotional intensity by using of the type of authentication.Experimental results show that this method can improve the accuracy of the theme microblog's emotional strength increased by 0.05?0.1.(3)This thesis implements the microblog public opinion analysis system and the web display,and gives experimental analysis for the microblog data,such as the time distribution of publishing behavior,the time distribution of microblog forwarding delay and the hotspot of microblog public opinion.Experimental results show that the system can monitor microblog public opinion in real-time combines with proxy collecting system and can effectively analyze the emotional tendencies of public's opinion hotspot with the optimized method of the microblog's emotional intensity.