Comparison of blind relevance feedback algorithms using controlled queries

One of the current issues with document retrieval is that it is not fully understood why some algorithms are better than others. Much of this confusion is due to evaluations typically being done on a set of user-defined queries. Such evaluations have little control over the amount of information in the query. Without this control it is difficult to know what are the causes of a given retrieval performance. Proposed here is a new evaluation method that addresses this lack of control; algorithms are tested on queries that are autonomously generated from predefined relevant sets of documents. Relative entropy is used to discover the most discriminating terms which are used to manufacture these queries. In this work, two blind relevance feedback (BRF) approaches are evaluated using these controlled queries. The results indicate that BRF does not improve retrieval performance for queries composed of only the most discriminating terms.