Deep Recursive And Hierarchical Conditional Random Fields For Human Action Recognition

Recognizing human activities from videos is an essential issue in computer vision and pattern recognition due to its significant applications in areas such as video surveillance and retrieval. Recently, inexpensive RGB-D cameras(such as Microsoft Kinect, etc) have enabled rearchers to model the rich spatio-temporal interactions in the 3D scene for learning complex human activities.Most previous work on activity Recognition has focused on using 2D videos. The use of 2D videos leaded to relatively low accuracy even when there is no clutter. In this work we perform activity recognition using an RGB-D sensor and recognize a person’s activity by the cooccurence and geometric constraints of human pose and object interacted in activity. We use a rigid skeleton to describe a person and using the extracted skeleton model to find the bouding boxes for the person’s parts, computer HOG-3D features for each of these bounding boxes as the human pose features. At last, we concatenated the human pose features, object features, object-subject relation features, etc into a single feature vector, which is considered as the observation of one activity segment.The Linear-chain CRFs is one of the most popular discriminative models for human action recognition, as it can achieve good prediction performance in temporal sequential labeling by capturing the one- or few- timestep interactions of the target states. However, existing CRFs formulations have limited capabilities to capture deeper intermediate representations within the target states and higher order dependence between the given states, which are potentially useful and significant in the modeling of complex action recognition scenarios. To address these issues, we formulate a deep recursive and hierarchical Conditional Random Fields(DR-HCRFs) model in an infinite-order dependencies framework. The DR-HCRFs model is able to capture richer contextual information in the target states, and infinite-order temporal-dependencies between the given states. Moreover, we derive a mean-field-like approximation of the model marginal likelihood to efficiently facilitate the model inference.The parameters of the predefined model are learnt with the Cutting-Plane algorithm, Frank-Wolfe algorithm and block-coordinate primal-dual Frank-Wolfe algorithm in a structured support vector machine framework respectively. Experimental results on the CAD-120 benchmark dataset demonstrate that the proposed approach can achieve high scalability and perform better than other state-of-the-art methods in terms of the evaluation criteria.