Human Action Segmentation And Recognition Based On Depth-sensing Camera

As a key step in an overall human action understanding system, action recognition is applied for many common applications including human computer interaction, video surveillance, game control system and etc. Compared with the traditional 2D informa-tion, the depth information captured by depth camera has obvious advantages because the z-index displacement information which is lost in 2D frames is valued here.On the other hand, action segmentation appears to be a problem no less difficult than action recognition itself. We can get satisfactory accuracy by training and testing on segmented clips, but the effect is hard to maintain on the real-time video stream and synthetic consecutive action clips.In this paper, we present some methods to automatically separate consecutive human actions into subsegments and recognize them. The body skeleton positions tracked by depth camera and the depth images are used in the method. So we com-bine the pairwise relative positions of the Skeleton Joints, Depth Motion Maps(DMM) and DMM-Pyramid computed from the depth images together to improve the feature representation. SVM,2D/3D convolutional neural network(CNN) and recurrent neural networks are used as classification ensembles. For action segmentation, we build a Probability-Distribution-Difference (PDD) based dynamic boundary detection frame-work and a maximum-subarray-search based temporal smoothness method. The seg-mentation methods are both online and reliable.The experimental results applied to the MSR Action3D dataset and so on outper-form the state of the art methods.