Research On Human Action Recognition Based On Graph Convolutional Neural Networks

The main goal of human action recognition is to enable the machine to automatically recognize human motions from images or videos.It has broad application prospects in video surveillance,healthcare,smart home and human-computer interaction.Since the human body can be regarded as an articulated system in which the joints are connected by rigid joints,the motion is mainly reflected in the skeleton motions in the 3D space.Due to the skeleton information is robust to changes in scale,illumination and viewpoint,skeleton-based action recognition has become a topic of concern in the field of computer vision and pattern recognition research.In this dissertation,we focus on the three key issues of skeleton-based action recognition,i.e.,(1)robust spatial feature extraction,(2)temporal motions modeling and(3)motion-related part capture.To this end,we carry out extensive researches based on graph model theory and deep learning method.Concretely,this dissertation has the following three main contributions:(1)We propose a graph convolutional neural network inspired by the attention mechanism,which can effectively extract robust spatial feature and capture motion-related part.Firstly,in order to extract the deep features from the irregular skeleton data,we use graph to represent the skeleton data,and introduce the spectral filtering to realize the efficient graph convolution operation flexibly.Secondly,to adaptively detect the salient action units,we design a novel action-attending layer,which also helps to extract discriminative features.Finally,we use a recurrent neural network unit for modeling temporal motions.Thus,we propose an end-to-end deep neural network.(2)We propose a spatio-temporal graph convolution model inspired by the autoregressive moving average model,which can encode the spatial structures and dynamic patterns simultaneously.Skeleton-based action recognition can usually be regarded as a temporal problem.To achieve accurate and reliable action recognition,the spatial and temporal dependencies in skeleton data should be well modeled.For this motivation,we design the multiscale graphical convolutional kernels for encoding the graph-structured data.Inspired by the autoregressive moving average,which is skilled in processing time series problems,we design the recursive graph convolution model.In addition,we prove the stability of the proposed model,provide a theoretical upper bound,and analysis the influence of graph convolution kernel size and model configuration experimentally.(3)We propose two kinds of spatio-temporal graph convolution models combined with recurrent neural networks,which can encode the temporal motion patterns of skeleton graph nonlinearly.The spatio-temporal graph convolution proposed in work(2)belongs to a linear model essentially,which limits the performance.In order to achieve complex function approximation and enhance the fitting ability of the model,we further extend it to the nonlinear dynamic networks,which take the philosophy of long short-term memory and gated recurrent unit.These models not only inherit the success of local convolution filtering,but also achieves the ability of sequence modeling of recurrent neural networks.In addition,these models can be used as a basic layer to construct deep networks.To evaluate the proposed models in this dissertation,we conduct extensive experiments on four benchmark skeleton-based action datasets and compare with the state-of-the-art methods.The experimental results demonstrate the effectiveness of our proposed models and a more promising direction for skeleton-based action recognition.