| The objective of this study was twofold: first, to describe the criteria mentioned by users evaluating the information within documents as it related to the users' information need situations; and second, to describe the characteristics of documents that allowed users to predict the presence or absence of criteria within documents. This research was intended to provide a better understanding of the notion of relevance, which is a fundamental concept in the design and evaluation of information retrieval systems.; This study was an exploratory effort to examine these issues by eliciting mentions of criteria and document characteristics directly from motivated users in actual information seeking and use situations. Data were collected by asking users in an academic environment (e.g. undergraduate students, graduate students and faculty) to evaluate representations of documents that had been retrieved specifically for each user's information need situation. Users were asked to mark the portions of the document representations that indicated to the users whether they would or would not pursue the information within documents. An open-ended interview technique was then employed to discuss each marked portion of the documents with users. The interviews were audiotaped, the tapes transcribed, and the transcriptions were content analyzed in order to identify and describe evaluation criteria and document characteristics.; The major results consisted of descriptions of document characteristics (e.g., authors, publication dates) and criteria (e.g., recency, accuracy) mentioned by users. The results indicated that the criteria employed by users included tangible characteristics of documents (e.g., the information content of the document, the provision of references to other sources of information), subjective qualities (e.g., agreement with the information provided by the document) and situational factors (e.g., the time constraints under which the user was working). The results also examined the co-occurrence of document characteristics and criteria, in an attempt to determine where users were finding clues to the presence or absence of criteria. The implications of this research for the design and evaluation of information retrieval systems, as well as directions for future research, are discussed. |
