| The performance of information retrieval systems varies significantly by test topics. Even for those systems that have performed well on average, the results for some difficult topics are still poor. Previous studies have revealed that different optimization techniques should be used for those difficult topics. However, a prerequisite of the discriminative treatment is to assess the systematic difficulty of a topic without relying on any external relevance judgment. This study surveyed and selected a number of the most popular existing predictors from the literature for this task, proposed two new predictors based on the classic Vector Space Model, examined the performance of the individual predictors, investigated the performance of the combined predictors with a multiple regression model, tested the effect of incorporating a topic model to the task, discussed the factors of different similarity measures and different term weighting schemes, and compared the two most influential retrieval models for this specific task. To summarize the key findings: low to medium level correlations were found for the individual predictors. One of the newly proposed predictors has comparable performance with the best existing ones, while the other new predictor has special application in predicting the precision at the top cutoffs. Combining the predictors with a multiple regression model showed improved results. A number of techniques that can be used to improve the performance were further examined: a topic model, a query dependent similarity measure and tuning the term weighting function. The study is built on the foundation of many previous efforts and adds valuable insights to the current knowledge base. The results and findings from this study provide a comprehensive understanding of the existing problems, possible solutions and future trends for research of assessing systematic topic difficulty. |
