Design And Implementation Of Theme Crawler Oriented Micro-blog

The rapid development of Internet makes the Internet has penetrated intopeople’s daily life, in all aspects of society are affected. A survey of authority, thenumber of Internet users in the rapid increasing every year, and the number of netusers in China has ranked the first in the world. Internet users can speak one’s mindfreely, in the Web communicate with each other, so the Internet has become a virtual "public discourse space ", is a new public opinion platform the public can participate inthe.With the rapid development of the Internet, micro-blog has become contains avariety of information resources, the site distribution of global information servicenetwork. Micro-blog crawler is a used to help Internet users to query informationsearch tool, it has a strategy to gather in the micro-blog, find information,understanding of the information, extraction, processing and organizations, and toprovide search services and information navigation for users.The main purpose of this project is to design subject-oriented micro-blog crawlerprocess which is also required to meet certain performance, taking into account thediverse needs of micro-blog crawlers.The main purpose of this project is to design-oriented microblogging themecrawlers,also need to meet the performance and functional requirements,taking intoaccount the different needs of reptiles microblogging, reptile microbloggingapplication width search technology, analysis of the URL,remove duplicate.Microblogging reptiles implemented using multi-threading technology, the reptilescrawling with better performance, network connections and network settings toconnect reptiles and reading time setting, avoid waiting unrestricted. In order to meetdifferent needs, microblogging use pre-set theme reptiles, to achieve a specific topiccrawling.This paper studies the principles of micro-Bo reptiles, And to achieve reptilerelated functions.