Decision-theoretic reminder systems that learn from feedback

I developed a decision-theoretic model of unsolicited information retrieval (UIR). The model, called DT-UIR, extends two existing approaches: a decision-theoretic model of document retrieval from the field of information science, and a rule-based reminder system from medicine.; To implement DT-UIR in a medical context, I defined document utility as the difference between the value of information in a document, which was measured by domain-expert assessment of the expected utility of changes in patient care plans caused by information, and time cost, which was measured as the utility of time physician's spent with the document. I implemented many variants of the model including ones that assumed negligible time cost (PRETRIEVE), and ones that assumed linear time cost (PRETRIEVE-TC). I developed a test collection for system evaluation by giving documents to physicians who were formulating management plans for real patient cases.; Using a leave-one-out design, I tested how well each system predicted the utility of documents in "new" situations. Using an evaluation metric similar to ROC-curve analysis, I tested how well each system arranged a set of documents in order of expert-judged utility. I compared systems with each other and with a comparable rule-based reminder system. I tested variants of the DT-UIR model that varied systematically by type of evidence used by the system, by the level of discretization of variables in the system, and by the use of data other than utility as the basis for retrieval decisions. I constructed and tested a hybrid reminder system that used expected utility to order a set of rule-selected documents.; I found that utility prediction was poor; causes included limited training data, and limited evidence from which the retrieval decision was based. The document orderings of all DT-UIR systems, however, were substantially better than random, although somewhat less than ideal. Variants using relevance feedback ordered documents less well than those using utility feedback. The orderings and utility of retrieved sets of documents of the PRETRIEVE-TC system were significantly better than those of the rule-based system.