| As a basic foundation and important component of a variety of services based on Web, Information Crawler is applied in many fields, such as search engine, site structure analyzing and web graph evolution. But because tremendous Web information has a dynamic, heterogeneous, high-repetition, high-link and multilingual characteristics, Information Crawling is a tremendous challenging task.Parallel is the good way to improve efficiency of crawling. In this article, we study how to design a good distributed framework for parallel crawler, and implement such a system called InsunSpider with robust and high scalability quality. To reduce design complexity of communication module, we adopt NFS(network file system) to intercommunicate between nodes. Meanwhile, communication load is reduced. And in this article, we use two-stage mapping model to implement dynamically reconfigurablity, then when there are nodes to join or detach, performance of system will not be affected. And there is a server in our system; its responsibility is to monitor and adjust. Some data of all nodes will be adjusted, for example, logical table and hash table when system changes.For updating, we propose an efficient algorithm for incremental crawling. It improves efficiency because this algorithm focuses on part of the Web based on characteristics of highly sensitive web pages and temporal locality in the change. Our experiments show that it is applied to small and medium-scale incremental crawling when limited to news website.In addition, we also research the technology of focused crawling, design and implement a module of topic filter based on Vector Space Model.Through the experiment results of analysis, we'll note that our system——InsunSpider have many excellent characteristics, for example, efficient, high scalability, high configurability, load balancing and so on. Currently, this system is providing data services for retrieval system and QA extraction. It is proved that our system has broad prospects and good practical value. |
