| Collecting relevance assessments is a very important procedure in Information Retrieval. It is conducted to (1) evaluate the performance of an existing search engine, or (2) build and train a new one. While most of the popular performance evaluation measures and search engine training algorithms assume the relevance assessments are accurate and reliable, in practice this assumption is often violated. Whether intentionally or not, assessors may provide noisy and inconsistent relevance judgments potentially leading to (1) wrong conclusions about performance of a search engine, or (2) inefficient or suboptimal training of a search engine.;Addressing the problem above, we first (a) demonstrate how one can quantify the negative effect of assessor disagreement (including intra-assessor disagreement as a special case) on the ranking performance of a search engine. Beside this theoretical result, we also propose practical recipes for (b) tuning existing evaluation measures with the goal of making them more robust to the label noise, (c) improving the reliability of relevance estimates by collecting and aggregating multiple assessments (potentially through crowdsourcing ), and (d) incorporating noise reduction component into learning-to-rank algorithms. |
