| Conventional evaluation of information retrieval (IR) systems is conducted empirically. Common metrics such as precision and recall are good performance indicators yet they fail to evaluate the functionality of the underlying IR models. Recently, inductive evaluation approaches have been proposed to circumvent this problem. In this thesis, we propose an "aboutness" based inductive evaluation for functional benchmarking of IR models. It is complementary to the traditional experimental based performance benchmarking (e.g. TREC). IR is driven by a process which determines the aboutness relationship between a document and a query. Recent attempts spawned from logic-based information retrieval theory have been made to formalize aboutness properties. Different IR models support different sets of aboutness properties. These sets enable the comparison of functionalities of various IR models from a theoretical viewpoint. Nonetheless, existing aboutness frameworks are ineffective for functional benchmarking. The proposed aboutness properties are largely determined by the framework, where the aboutness model is defined. Further, some aboutness properties are only sound within the context of a given IR model, yet they may not be sound from the perspective of the user. To overcome these predicaments, we investigate aboutness from a commonsense point of view, i.e. what are the fundamental aboutness properties acceptable from a human reasoning perspective, independent of any given IR model. We first adopt the most fundamental aboutness framework, namely the one proposed by Bruza, as the basis in defining initial functional benchmarking suite. We apply the suite to various IR models. We then re-assess the effectiveness of Bruza's framework for benchmarking and observe the advantages and deficiencies of the IR models under examination. Based on these observations and reassessments, we refine the existing notion of aboutness and propose a new commonsense aboutness theory. To illustrate the potential of this new theory, we apply it to functional benchmarking. The results can suggest how an IR model be improved to support desirable commonsense aboutness properties. The commonsense aboutness theory would lay down a significant theoretical groundwork in IR research, which could (if properly deployed) lead to more effective IR systems. |
