Research And Implementation Of Result Merging In Distributed Search

With the rapid development of the Internet, both quantities and information richness ofweb pages grow very fast, and the information resources are becoming more and moredistributed. The situation above brought a lot of challenges to the traditional centralizedsearch engine, especially on system’s scalability, retrieving the deep web and search resultsdiversification. Distributed search engine would be a suitable solution to meet the requirementof the structural features and potential trends of the information’s distribution in NextGeneration Internet. Based on scalable distributed architecture, distributed search engine canprovide users with a unified retrieval services while integrating the distributed informationresource and using them effectively.This paper studies federated search system of the distributed search engine based on thenational next-generation internet CNGI project “Next Generation Distributed Search Engineof the Internet”. Federated search system is the core module of the distributed search engine.It can automatically forward the query to single search engine (called unit search engine) andmerge the returned results from unit search engines. Query forwarding and result merging aretwo key technologies of the federated search system. This paper aims to study the suitablestrategies of query forwarding and result merging.This paper roots on the real data set from campus network, and uses resource’s score toevaluate the relevancy between queries and unit search engines. Resource’s score isdetermined by static and dynamic features extracted from the unit search engines. Then thispaper proposes the strategies of query forwarding based on the resource’s score, which wouldensure the quality of the return results. The optimized ranking algorithm of results mergingincludes three processing steps: normalization of result documents’ scores, result mergingalgorithm based on the query forwarding, and finally result merging mechanism whichemphasizing results diversification. Some experiments have been done on the system, and theexperimental results demonstrate that the strategy of query forwarding based on resource’sscore and the optimized ranking algorithm of results merging can improve the precision ratioon the system’s retrieval result, and also enhance the effect of results diversification.