Human Activity Recognition in Video: Extending Statistical Features Across Time, Space and Semantic Context

This thesis explores the problem of recognizing complex human activities involving the manipulation of objects in high resolution video. Inspired by human psychophysical performance, I develop and evaluate an activity recognition feature derived from the velocity histories of tracked keypoints. These features have a much greater spatial and temporal range than existing video features. I show that a generative mixture model using these features performs comparably to local spatio-temporal features on the KTH activity recognition dataset. I additionally introduce and explore a new activity recognition dataset of activities of daily living (URADL), containing high resolution video sequences of complex activities. I demonstrate the superior performance of my velocity history feature on this dataset, and explore ways in which it can be extended. I investigate the value of a more sophisticated latent velocity model for velocity histories. I explore the addition of contextual semantic information to the model, whether fully automatic or derived from supervision, and provide a sketch for the inclusion of this information in any feature-based generative model for activity recognition or time series data. This approach performs comparably to established methods on the KTH dataset, and significantly outperforms local spatio-temporal features on the challenging new URADL dataset. I further develop another new dataset, URADL2, and explore transferring knowledge between related video activity recognition domains. Using a straightforward feature-expansion transfer learning technique, I show improved performance on one dataset using activity models transferred from the other dataset.