The Design Of Specific Topic Web Crawler And Its Transmission Group

With the rapid development of search engines, web crawlers which play a keyrole in them have also achieved a great improvement. Hereinto, the specific topic webcrawler becomes more and more attractive. On the other hand, with the burst ofmobile internet, more and more social network applications come out. For example,microblog and WeChat are multiplied and the news media and governmentannouncements also fall into short texts. Facing the forum, microblog and WeChatlike short texts, traditional methods rely on the search keywords provided by theirown company. However, when the users meet the large amount of texts, to solve theinformation explosion, it needs to select special topic to search information. In thispaper, based on the work requirements we design and implement a web crawler forthe specific topics.This paper firstly introduces the relevant knowledge for search engine and webcrawler. After the analyzing and comparison for the common search strategies andweb crawler algorithms, we show the detailed discussion for the specific topic webcrawler. Aiming to obtain data for the particular Web pages, we employmeta-search-related technologies. To study the spread range of special topics, weintroduce micro-blog data. To sufficiently use the social network of social media, afterobtain the node of people, we can briefly understand the spread crowd and study theirdistribution in social network.