A Bias-Variance Evaluation Framework For Information Retrieval Systems

The information retrieval evaluation metrics include the effectiveness metrics and the stability metrics.However,most of these stability metrics and effectiveness metrics are defined separately.Researchers found that the improvement of mean retrieval effectiveness may sacrifice the retrieval stability over topics,indicating that there is a trade-off between effectiveness and stability.Besides,the relation between effectiveness and stability has not been systematically studied.Traditional information retrieval evaluations are based on a fixed document collection,meaning that the effectiveness and stability of the system is based on a set of topics.When the document collection is a sample from a larger document collection population,the effectiveness and stability of the system on per-topic and the impact of the document collection being sample have not been fully investigated.In addition,the relation between significance test and the bias-variance decomposition evaluation is explored in this paper.Based on the bias-variance decomposition theory of mean squared error,the biasvariance evaluation framework of information retrieval system is proposed.The evaluation framework we proposed can not only evaluate the stability and effectiveness of the system separately,but also evaluate the overall performance(i.e.,effectiveness and stability)of the system.The relation between effectiveness and stability is also effectively studied under this framework.Experimental results on Ad-hoc track and Web track show that there is a trade-off between the effectiveness-stability.Experiments on Ad-hoc show that per-topic variance has no significant effect on the comparisons of system ranking,but system comparison based on significance testing are more susceptible to per-topic variance.The experimental results on the Session track show that query modification and more user data can simultaneously improve the effectiveness and stability of the system.