Research And Implementation Of Public Opinion Data Crawl System

With the rapid development of Internet,people are more dependent on the network as it becomes more common and applied in life widely.It is not doubtful that Internet is such a huge database which contains prodigious amount of information classified in various categories.However,people are only interested in a little part of that information which makes public opinion become the topic of focus.As a result,achieving relevant information efficiently and promptly should be a significant technology.This research is mainly on public opinion data crawl function demand,implementation of the technology of data system of crawl and other basic analysis functions.This system can be used to crawl public opinion and hot spot data then analysis them and produce the final result.System functions are divided into three parts,including public opinion,hot spot module and monitoring.The part of the public opinion is mainly at public opinion data template management,public opinion data acquirement and public opinion information searching.Hot spot module is mainly at hotspot data template management,hotspot data fetching and hot analysis sorting.The part of monitoring is mainly to achieve site monitoring comparison and monitoring management..Focus crawler algorithm for hot crawl data is applied in this research to improve the crawl efficiency and correlation when operating the crawl for hot data and the bloom filter is used to avoid repetition for web URL.According to the running environment and function demand,data crawl function should adapt different use frequency and service at huge amount of data.Therefore,high crawl efficiency and low data repetition rate should be guaranteed during the crawl process and the stability,coordination and extensibility of this system should be achieved as well.Finally,a highly efficient and convenient public opinion data system will be presented.