| With the rapid development of Internet, network resources are more and more abundant andcolorful. This is a huge challenge for general-purpose information crawling system and searchengine. People’s demands for information services are more and more high and professional.Therefore general-purpose search engine can’t meet the demand for professional fields. In thiscase, focused crawler came into being. Now, IPv4address has been run out and IPv6isdeveloping. The number of IPv6address increased fast. Demands for IPv6resources are moreand more much. In this case, we need an IPv6information crawling system to fetch IPv6resources.This pager aims to design and implement an effective, robust, configurable, accurateIPv6information crawling system. This system provides IPv6resources for search engine tomeet people’s demand for IPv6resources. This paper introduces the present research situationon information crawling firstly. Then paper introduces some relative theory knowledge aboutsearch engine, including development of search engine, basic theory of information crawling,focused crawler and page analysis algorithm. This paper uses distributed system frame andMVC design model to implement IPv6information crawling system. The system improvessystem’s performance through DNS cache, robots cache and site information cache. Thispaper also comes up with some crawling strategies to guide the system to fetch IPv6resources,including education sites first, large site first and score transition based on site link structure.The system uses RMI technique to implement communication between distributed nodes.Master node sends execution commands to slaver nodes and slavers nodes send heartbeatinformation to master to report nodes’ status.This paper has carried out the system test on the following aspects: DNS cache effect test,system crawling performance test, IPv6crawling strategy. After crawling IPv6resources, thesystem will obtain and analyze topology structure and resources distribution of IPv6sitesthrough statistics of sites information. |
