A Study Of Risk And Evaluation In IR Model

With the development of Information Retrieval technology,the risk problem in different stages of IR model,such as risk in relevance estimation,document ranking and query expansion.However,it is difficult to model the risk in IR because of its diversity.So we first focus on designing an evaluation method to evaluate the performance mean value and the risk of IR model,and then find out the strategies to reduce risk.In this paper,we study two important problems.One is a unified formulation based on the bias-variance decomposition to evaluate the retrieval effectiveness and stability.Furthermore,we generalize the metric to different effectiveness evaluations and their means and design an unbiased target model for the metric.The other is how to reduce the risk in query expansion.In this paper,we aim to explore the utilization of the knowledge about query-related entities and their properties in Freebase as a solution for the improvement of the retrieval effectiveness and stability of query expansion.The relevance score between each property term and the query is measured by the risk-reward analysis in portfolio theory,which treats property terms as a whole set and is expected to maximize the reward of the relevance scores of property terms and minimize the risk of query expansion failure using these property terms.In the experiments,we first carry out the generalized bias-variance metrics over the runs in the TREC ad hoc and web track.We observe a clear effectiveness-stability tradeoff in the historical runs.Then we evaluate the retrieval performance of the query expansion model based on Freebase on two web collections,the results show that the expansion model performs more effectively on reducing risk in query expansion.than the relevance model(RM3).