Research On Focused Search Engine Based On Mobile Agent

Due to the current large size and dynamic of the Web, gerneral purpose search engines can no longer provide a comprehensive and up-to-date search service of the Web.In contrast to gerneral purpose search engines, which attempt to index the whole Web, Topic-Specific search engines can cover specialized topics in more depth and keep the crawl more freshly. The design and implementation of a Topic-Specific search engine is going through a highly creative phase.Internet is becoming to be the largest information base although the Web is full of a mass of pages. It is the great problem facing to the information scientists that how to make people obtain the information quickly and accurately.In this paper, we firstly analyzed the development of search engine, described the implement of each part of search engine, based on this, investigated the background of research on focused search engine, analyzed the reasons of why research on focused search technology is booming, and evaluated the potential value of focused search. On the basis of reading a lot of references, introduced some typical focused search engine project.Analyzed many important technologies used in focused search that include: the conception of topic, the technologies of finding topic resource, the construction of focused crawler, heuristic strategies of focused crawling and related algorithms. According to the features of focused search engine and mobile agent technology, a method that use mobile agent works on remote host to retrieve topic information has been brought forward. Discussed how this method can improve the performance of focused crawling, especially at the aspect of reduce the cost of network bandwidth. Brought forward a prototype of mobile agent based focused search engine, and described each part of the system. Implemented some core parts of the system, include: distilling of topic character, system architecture of mobile agent, the working manner of mobile agent, the crawling of topic-related pages and the parsing process of their content, the computing of relevance.Finally, on the IBM aglets platform, we developed a mobile agent based focused crawler. Through experiment, we made a comparison between the mobile agent based focused search and traditional search, result shows such agent-based focused search can reduce the amount of retrieved pages that little relevant to a topic that transferring in network, so that saved the unnecessary network band cost, and also improved the efficiency of topic information retrieval.