Research On Retrieval Algorithms For Non-cooperative Distributed Search Engines With Multiple Verticals

As the large scale and the diversity of the Internet,people prefer search engines for getting information.However,search engines are often with different verticals,which means that users may have to surf on different engines for the same query needs and often they can not get the most relevant results.Federated Search is the solution to cope with it.Among real federated search systems,most of them are in the non-cooperative environment with the collections and the information that the system get from the colletions are limited which is a challenge in federated search.In this paper we will focus on the research on federated search of noncooperative environment with multiple vertical information from the colections,including key algorithms of vertical selection,resource selection and result merging.As for the result,we propose serveral new frameworks.1)In the vertical selction,we propose a LDA-based method for constructing vertical description and query description.With the descpition constructing method,we test the vertical selection algorithm by nDCG@10 and nDCG@20 and the result shows the effectiveness of the method.2)In the resource selection,we propose a scoring framework with three important features,including LDA-based scores of resources,vertical selection results and the search engine impact factor.The results of the experiments prove that the merging method is better than every single feature scoring method of the three.3)In the result merging,we propose methods based on LR and GBDT with features of documents,resources and verticals.The experiments show that our methods perfrom equally with other algorithms in nDCG@10 and nDCG@20,and better in bringing down the complexity of tuning parameters.In the end of the paper,we have a conclusion of our methods and make our plans for the future research.