| When people use an information retrieval system, how many retrieved documents do they examine? How is this number affected by the quality of the information retrieved? This research explores these questions from two perspectives: the system's and the user's points of view. First, the performance of three commercial retrieval systems (DIALOG's Boolean retrieval, LEXIS-NEXIS's Freestyle, and DIALOG's Target) was compared at different cutoff points, i.e., when 10 documents were retrieved, when 20 documents were retrieved, and so on. The comparison was made using a database of 1,239 documents taken from MEDLINE, and its goal was to determine the capabilities of the retrieval systems that are currently available commercially, that is, when and how they perform best. The results showed that Freestyle had the best retrieval performance at the majority of the cutoff points.; Secondly, qualitative methods were used to observe and interview 115 health professionals as they searched MEDLINE in order to discover the number of documents they were willing to examine and any factors affecting user persistence. The median number of references that subjects stated they would be willing to examine was 100, but the median number of references actually examined during the searches was 41. Subjects mentioned a number of factors affecting their persistence, including: topic, purpose, type of search, search stage, search specificity, relevance of retrieved references, timeliness of retrieved references, time pressures, fatigue, and level of searching skill. Other factors that appeared to affect user persistence were searching experience, MEDLINE training, and education level.; The design of this study attempts to build on the strengths of the systems and the user perspectives by approaching the question of optimal document cutoff points (i.e., the optimal number of documents retrieved) from both points of view. Thus, the optimal cutoff points for three retrieval engines available commercially were determined and compared to the number of document representations that health sciences professionals are willing to examine in a real-life environment. This comparison indicated that, on the average, each system performed best at cutoff points that are less than the number of references the average user is willing to examine. |
