Human Action Recognition Based On Feature Representation And Attribute Mining

Human action recognition is a fundamental and challenging topic in the area of computer vision, which attracts more and more attentions. The applications of human action recognition contain smart surveillance, video content analysis, human-computer interaction, smart home and so on. Because of these promising and valuable applica-tions, many researches are focused on this topic in decades. In this topic, how to extract discriminative representation for action video, is the key to human action recognition. In this article, we work on human action based on feature representation and attributes mining. The target of this work is to explore the discriminative representations of hu-man action video from multiple aspects, including low-level features, action attributes and feature learning from deep networks. Based on the discriminative representations of action video, human action can be recognized by simple classifiers.In this thesis, we concentrate on feature representation and attribute mining for action video. We extract low-level features and mine action attributes from depth-based skeletal action videos and also describe traditional RGB action videos with multiple features, including local features and deep network features. The contribution of this thesis is threefold as follows.(1) For depth-based skeletal action videos, we design effective low-level feature representation, and adopt Markov Random Field model to encode features of human skeleton by considering the spatial and temporal consistences on human skeleton se-quences. This representation suppresses the intra-class variances and also noises in the skeleton data. We also propose the pattern-based M4IL to learn discriminative skeleton motions for each action class to build a low-latency human skeletal action recognition system. These learned skeleton motions can be used to measure the similarity between every skeleton with each action class.(2) For depth-based skeletal action videos, we propose an attribute-based skeletal action recognition and explore the scalable action recognition from two aspects, i.e., attribute sharing and attribute space expanding. We first present a new skeletal fea-ture with the representations of static pose and motion of human skeleton to support a comprehensive action attribute space. Then, a novel action attribute mining method is proposed to discover action attributes for each bone pair. Finally, we accomplish ac-tion recognition based on those mined attributes. The mechanism of discovering novel attributes is proposed to expand attribute space for scalable recognition.(3) For traditional RGB action videos, we track and crop human region from videos by dense trajectories and extract appearance representation based on deep Convolution Neural Network. We also extract the local motion representation along with dense tra-jectories. We use Hidden Markov Models to learn each feature type as a weak classifier and fuse these feature models by AdaBoost. We evaluate the performance of multiple feature representations for human action videos.This thesis focuses on the representation of human action, including depth-based human action videos and traditional RGB videos. We discuss this topic from three as-pects, i.e., low-level features, action attributes mining and multiple features fusion. In each aspect, we also consider the motivation from applications. For3D human action recognition, we seek the low-latency and scalable action recognition. For traditional RGB videos, we explore the theory and the time consumption of recognition model. We evaluate the proposed methods with extensive experiments to demonstrate the ef-fectiveness and superiority, which also inspires us for future work.