A Study Of Value At Risk For Risk Evaluation In Information Retrieval

The risk of information retrieval comes from three aspects : first,query is ambiguous,the relevance between query and document is uncertainty and the relevance between document in document ranking.Based on the sources of risk,many risk metrics have been proposed.But,they are just considering the ratio that the improved or hurt performance query number and all queries.Thus,they have some limitations,when two systems have the same performance score,while they have different document rankings.The existing risk metrics can't distinguish them from the value of risk.In this paper,we propose a novel risk evaluation named value at Risk to deal the limitation that the existing risk metrics have.Value at Risk is widely used in finance areas,we apply it into information retrieval named VaR_IR.It not only considers the performance score,but also take the relevance between documents in the ranking.After that,we also study the source of performance variance,and we analyze the variance from topic and system with mixed-effects model.The experiments show that the method we have proposed VaR_IR is a complementary to and can be integrated with the effectiveness metrics to provide a more comprehensive evaluation method.The experiments show that the topic and system can influence the variance of performance.