Research On Graph Convolutional Networks For Skeleton-based Action Recognition

Action recognition is a very important part and hot research topic in the field of computer vision.Because people can use action to deal with affairs and convey emotions,action recognition technology can also be widely used in different fields,such as security monitoring,human-computer interaction,video classification and so on.Because skeleton data can more effectively resist the transformation of viewpoint and motion rate in more complex background,skeleton-based action recognition has developed rapidly in recent years.On this basis,Graph Convolutional Network(GCN)has become a key research direction of scholars in the task of skeleton-based action recognition because its graph topology can more effectively model the dependency between skeleton.However,there are some problems for skeleton-based action recognition method based on GCN,such as insufficient extraction of local low-level motion features of skeleton,prominent information isolation of common skeleton data modalities,imperfect construction of graph topology and so on.To solve the above problems,this paper mainly studies the subject from three aspects: reconstructing the convolution mode of partial graph convolution layer,enriching and expanding input skeleton data and features,and enhancing network graph topology modeling.The main contributions and achievements of this paper are as follows:(1)This paper proposes a Two-stream Mixed Graph Convolutional Networks(2s-MGCN)based on exclusive kernel graph convolution.In view of the physical characteristics of skeleton data,in order to improve the extraction ability of low-level discriminative motion features of skeleton data,Exclusive Kernel Graph Convolution(EKGC)is proposed.In the convolution operation of EKGC,more specific motion patterns of each skeleton region are extracted by assigning exclusive convolution kernels to each graph node.At the same time,due the respective characteristics and advantages of Exclusive Kernel Graph Convolution and Regular Graph Convolution(RGC),a mixed graph convolution strategy is proposed.The EKGC module is used in the shallow layers of graph convolution network and the RGC module is used in the deeper layer.Then,a Mixed Graph Convolutional Network(MGCN)is proposed and constructed.Finally,the score fusion strategy is used to fuse the prediction results of joint data and bone data,so as to get the final prediction prediction of human action classification.On the NTU-RGBD dataset and Kinetics-Skeleton dataset,the performance of the algorithm proposed in this paper is compared with other relevant advanced algorithms,which proves the effectiveness and competitive advantage of the algorithm.(2)This paper proposes a Multi-stream Mixed Graph Convolutional Networks(MSMGCN)with angle data.In order to fully mine and use the complementary effect between multi-modality skeleton data,in addition to calculating and extracting the joint data,bone data and motion data of the original skeleton data,the angle data of skeleton and its extraction algorithm are also proposed.Skeleton angle data can show good robustness against the change of view angle and size of skeleton data.Compared with the three existing skeleton data,the angle data realizes the expression and construction of skeleton correlation and dependence at the data level.With the help of MGCN as the basic tributary network,the proposed MS-MGCN model can combine the skeleton features of multiple modes,and fuse the prediction scores of various modality data streams,so as to draw the final prediction result.Experiments on NTURGBD dataset,NTU-RGBD120 dataset and Kinetics-Skeleton dataset confirm that the algorithm presented in this paper has competitive advantages over other advanced skeletonbased action recognition algorithms.(3)In this paper,a Multi-stream Motion Enhancement Graph Convolutional Networks(MS-MEGCN)is proposed.In order to use the motion data of skeleton to strengthen the dependency between the active skeleton in the action and the overall skeleton,this paper proposes a measurement and calculation method of skeleton motion intensity to quantify the motion intensity of each skeleton in the action,constructs the motion intensity graph according to the skeleton motion intensity,combines the specific refined topology of the channel and the self-decay natural connection praph proposed in this paper to model the dependency of the skeleton,Motion Enhancement Graph Convolutional Network(MEGCN)is proposed to enhance the motion feature extraction ability of the model.At the same time,in order to overcome the information isolation of existing skeleton data,a cooperative action feature constructor is proposed to construct a cooperative action representation feature between skeletons under a specific paradigm.As a high-level feature that can be obtained without graph convolution,the cooperative action feature depicts the dynamic cooperative relationship between skeletons on different scales,ranges and distances.This feature can be used as a supplement to the existing skeleton data and fed to the same graph convolution network for higher-level action feature extraction.The final network is based on the high-level cooperative motion characteristics obtained from the input joint data and bone data,and then the network topology is established according to the motion data.Then,together with the joint data,bone data and motion data,it is fed to each tributary MEGCN,and the final fusion prediction result is obtained.Experiments are carried out on two large skeleton-based action recognition databases NTU-RGBD and NTU-RGBD120,and the experimental results are compared with a variety of skeleton-based action recognition algorithms,which proves the effectiveness of the MS-MEGCN model proposed in this paper.To sum up,this paper studies graph convolution network for skeleton-based action recognition,proposes three skeleton-based action recognition networks: 2s-MGCN,MSMGCN and MS-MGCN,and carries out experiments on multiple skeleton datasets to verify the excellent performance of the algorithm proposed in this paper.