| Supervised learning deals with training models using seen examples to make predictions for the unseen ones. For instance, given example documents that are pre-annotated as personal, work, family, etc., a machine learning algorithm can be trained to sort new documents into folders automatically. To train an effective model, the algorithm needs to see many training examples (e.g., documents and their categories). Obtaining these examples often involves consulting a human expert whose time is limited and valuable. Active learning algorithms select which examples would be most cost-effective for consultation with the human.;Typical active learning algorithms simply choose the examples that should be asked to the expert. In this thesis, we take this one step further: we make better use of the expert's time by showing not the full example but only a short and relevant snippet of it, so that the expert can provide the answer faster. The downside is, however, if the snippet is too short or is irrelevant for the task at hand, the expert might not be able to return an answer at all. Similarly, if the snippet is misleading when taken out of context, the expert might err and return an incorrect answer. Therefore, the algorithm needs to choose which and how long of a snippet to show to the expert to simultaneously minimize expert time and maximize expert response rate and correctness. We refer to this approach as anytime active learning.;In this thesis, we focus on three aspects of anytime active learning: i) assuming the beginning of a document is most relevant, the algorithm has to decide where to truncate the document, ii) given a document, the algorithm optimizes for both snippet location and length, and lastly iii) the algorithm chooses not only the snippet location and size but also which documents to choose snippets from so that the snippet length, the correctness of the expert's response, and the informativeness of the document are all optimized in a unified framework. |
