Human Action Recognition Via Fused Kinematic Structure and Surface Representation

Human action recognition from visual data has remained a challenging problem in the field of computer vision and pattern recognition. This dissertation introduces a new methodology for human action recognition using motion features extracted from kinematic structure, and shape features extracted from surface representation of human body. Motion features are used to provide sufficient information about human movement, whereas shape features are used to describe the structure of silhouette. These features are fused at the kernel level using Multikernel Learning (MKL) technique to enhance the overall performance of human action recognition. In fact, there are advantages in using multiple types of features for human action recognition, especially, if the features are complementary to each other (e.g. kinematic/motion features and shape features). For instance, challenging problems such as inter-class similarity among actions and performance variation, which cannot be resolved easily by using a single type of feature, can be handled by fusing multiple types of features.;This dissertation presents a new method for representing the human body surface provided by depth map (3-D) using spherical harmonics representation. The advantage of using the spherical harmonics representation is to represent the whole body surface into a nite series of spherical harmonics coefficients. Furthermore, these series can be used to describe the pose of the body using the phase information encoded inside the coefficients. Another method for detecting/tracking distal limb segments using the kinematic structure is developed. The advantage of using the distal limb segments is to extract discriminative features that can provide sufficient and compact information to recognize human actions. Our experimental results show that the aforementioned methods for human action description are complementary to each other. Hence, combining both features can enhance the robustness of action recognition. In this context, a framework to fuse multiple features using MKL technique is developed. The experimental results show that this framework is promising in incorporating multiple features in different domain for automated recognition of human actions.