Vision-based recognition of actions using context

In this dissertation, we address the problem of recognizing human interactions with objects from video. Methods for recognizing these activities using human motion and information about objects are developed for practical, real-time systems. We introduce a framework, called ObjectSpaces, that sorts, stores, and manages data acquired using low-level vision techniques into intuitive classes. Our framework decomposes the recognition process into layers, i.e., a low-level layer for routine hand and object tracking and a high-level layer for domain-specific representation of activities. Segmenting recognition tasks and information in this way encourages model reuse and provides the flexibility to use a single framework in a variety of domains.;We present several ways of using context to aid in recognition problems. We exploit object context information (class, location, etc.) to help recognize hand-based actions and to enhance hand tracking. To classify unknown objects, we evaluate action context along with low-level image features and associations with other objects, where available. Throughout these approaches, we combine stochastic methods for classification of complex activities. Low-level hand actions associated with objects are recognized using the hidden Markov model. Markov chains are used to characterize a sequence of these low-level interactions so that single-tasked activities can be classified. At the very highest level, we use stochastic context-free grammar to represent the structure in activities that involve multiple objects and people over extended periods of time. We also provide extensions to the Earley-Stolcke parsing algorithm that enable error detection and recovery as well as adapt stochastic grammar to improve recognition. We also present methods of quantifying group and individual behavioral trends in activities with separable roles.;We show results of activity recognition in various domains, including an automobile, a kitchen, and an office. Our approach is appropriate for locating and classifying both objects under a variety of conditions including partial or full occlusion. We also provide results where both familiar and previously unseen objects are classified from action alone. From experiments with the card game, Blackjack, we produced high-level narratives of multi-player games and successful identification of player strategies and behavior.