Action Recognition Based On Depth Image And Skeleton Data

Human action recognition is a research hotspot in computer vision with the rapid development of computer technology.Related behavior recognition technologies have been widely used in surveillance,video games,human-computer interaction,and medical care.Early human behavior recognition methods are based on RGB images,but many factors such as light affect the recognition results.Depth images and skeleton data obtained by depth cameras can solve problems such as lighting.Depth images and skeleton data are not sensitive to changes in the background.The method of human behavior recognition using the above data is becoming increasingly popular.The depth camera obtains RGB images,depth maps and extract skeleton data.In order to improve recognition rate by getting more motion information from depth maps and skeleton data,the research about human action recognition based on skeleton data and depth images is conducted.The following two human action recognition algorithms based on skeleton data and depth images are proposed.The main works of this paper are as follows:Firstly,an action recognition algorithm based on depth images and skeleton data is proposed.The motion history point cloud(MHPC)is generated from the depth image sequence of an action and we extract global features from MHPC.At the same time,the relative displacement features and relative distance features are extracted from three-dimensional skeleton data and normalized them to generate a whole skeleton features by vector of locally aggregated descriptors(VLAD).Feature fusion is performed on the obtained point cloud features and skeleton features to construct a multi-modal feature fusion action recognition method.Experiments on MSR-Action3 D and UTKinect-Action3 D dataset prove that the method has better recognition effect for more complex actions.The recognition rate of the existing methods is improved in the set of cross-testing.Secondly,an improved Two-stream convolutional network based on skeleton energy maps and depth motion maps is proposed inspired by the Two-stream Convolutional network.The view-invariant transformation based on the skeleton sequence is used to eliminate the effect of view change.The transformed skeleton action sequence is visualized as a series of color images.The obtained color images are superimposed to obtain a skeleton energy map.Skeleton energy maps are further enhanced using image erosion methods.Add a deep motion map input stream of human motion to form a new dual-stream convolutional neural network framework to extract the spatiotemporal information in continuous motion frames.Experiments on the Northwestern-UCLA and UTKinect-Action3 D datasets prove that the method has better recognition effect.