Human Action Analysis And Recognition From Image Sequences

Human action analysis and recognition is a hot topic in the domain of computer vision and pattern recognition, and has promising applications to intelligent surveillance, visual reality and motion analysis. The key problems in this task are feature extraction, feature representation and action recognition. In this thesis, we focus on human action analysis and recognition from image sequences and investigate hand tracking, gesture recognition, human action and interaction recognition.This thesis proposes an algorithm for 3D hands tracking on the learned hierarchical latent variable space, which employs a Hierarchical Gaussian Process Latent Variable Model (HGPLVM) to learn the hierarchical latent space of hands motion and the nonlinear mapping from the hierarchical latent space to the pose space simultaneously. Nonlinear mappings from the hierarchical latent space to the space of hand images are constructed using radial basis function interpolation method. With these mappings, particles can be projected into hand images and measured in the image space directly. Particle filters with fewer particles are used to track the hand on the learned hierarchical low-dimensional space. Then the Hierarchical Conditional Random Field (Hierarchical CRF), which can capture extrinsic class dynamics and learn the relationship between motions of hand parts and different hand gestures simultaneously, is presented to model the continuous hand gestures. Experimental results show that our proposed method can track articulated hand robustly and approving recognition performance has also been achieved on the user-defined hand gesture dataset.Most researches on human action recognition are mainly based on the features of whole body motion. This thesis presents a hierarchical discriminative approach for recognizing human action based on limbs motion. The approach consists of feature extraction with mutual motion pattern analysis and discriminative action modeling in the hierarchical manifold space. HGPLVM is employed to learn the hierarchical manifold space in which motion patterns are extracted. A cascade CRF is introduced to estimate the motion patterns in the corresponding manifold subspace, and the trained SVM classifier is used to predict the action label for the current observation. The results on motion capure data prove the significance motion analysis of body parts, and the results on synthetic image sequences are also presented to demonstrate the robustness of the proposed algorithm.This thesis also explores a hierarchical approach for recognizing person-to-person interactions in an indoor scenario from a single view. It detects dense space-time interest points from action videos and divides them into two sets exclusively according to the history information and the connectivity of the two silhouettes. Then K-means clustering is performed on the combined set of interest points of all the training interactions to learn the spatio-temporal codebook. For a given set of interest points, a spatio-temporal word is built by allowing each point to vote softly into the few centers nearest to it and accumulating the scores of all the points. The CRF whose inputs are the spatio-temporal words is used to modeling the primitive actions for each person. Domain knowledge and first order logic production rules with weights are employed to learn the structure and the parameters of Markov Logic Network (MLN). MLN can naturally integrate common sense reasoning with uncertain analysis, which is capable of dealing with the uncertainty produced by CRF. Experiment results on our interaction dataset demonstrate the effectiveness and the robustness.