The Risk Sensitive Evaluation Based On Bias-Variance Decomposition

It has long been recognized in Information Retrieval(IR)that the improvement of mean retrieval effectiveness may sacrifice the retrieval stability across topics,indicating that there is a tradeoff between effectiveness and stability.Various risk metrics have been proposed to measure the retrieval stability.However,they were defined separately from effectiveness metrics.Previous work,such as risk-sensitive evaluation methods,consider effectiveness and stability simultaneously but do not systematically explore the effectiveness-stability tradeoff.In summary,there are three main problems in the field of information retrieval evaluation:(1)The effectiveness and stability of the system have not been systematically evaluated in a unified framework;(2)Under this framework,this unified metric can naturally be decomposed into effectiveness and stability;(3)The relationship between effectiveness and stability has not been systematically quantified.In this paper,we propose a new evaluation framework,based on the bias-variance decomposition of the machine learning,to measure the overall performance(considering both effectiveness and stability)and the tradeoff between effectiveness and stability of a system.Related experiments show that our framework can naturally combine effectiveness and stability,and the relationship between the two can be systematically quantified under the framework.In addition,experiments also show that our framework not only goes beyond the average effectiveness metric,but also includes more aspects of system performance,namely stability and the risk of the distribution of effectiveness scores against the baseline.