Action Recognition Based On Feature Fusion

In the past decades,human action recognition has been a hot topic in computer vision and machine learning.The main application fields of action recognition include abnormal behavior analysis,medical health monitoring,human-computer interaction,video monitoring and robotics.For many years,human action recognition mainly focus on RGB image sequence.Along with the development of imaging equipment,especially the appearance of low cost and high sampling rate Kinect camera,which can capture real-time depth and color images,depth image has been widely used.RGB image has abundant texture information,and depth image has better stability,because pixel values of depth images are not affected by illumination and environmental.In addition,the depth image can describe the three-dimensional information of the scene.This paper mainly studies action recognition from three aspects: multi-feature fusion of depth videos,two channel feature fusion of RGB and depth videos,and feature fusion of hand-craft and deep learning feature.The algorithm is described as follows.(1)Feature extraction of depth video sequences: Traditionally,in the research of human action recognition based on depth image sequence,the method based on depth motion map has achieved remarkable results.In this paper,we delete the beginning and ending five frame video sequences from video sequences,and project the rest of the depth frames into three orthogonal directions.In each projection direction,the absolute difference between two consecutive frames is accumulated to form a depth motion map(DMM).Based on DMM of three directions(front,side and top),image features are extracted,and human motion is described from three visual angles to form motion descriptors.(2)CNN features fusion of depth image and RGB image: Considering the Complementarity characteristics between depth image and RGB image,the features of RGB and depth images are fused into highly distinguishable fusion features.In this paper,a two-channel convolution neural network is used for depth image sequence and RGB image sequence to extract feature.In depth channel,a depth video sequence is first processed into a depth binary difference motion map(DBDMM)and input DBDMM into VGG19 Convolutional Neural Network for training and extracting features.In RGB channel,the background of each RGB sequence is removed based on corresponding depth Sequence,foreground RGB depth map(RDM)and the RGB difference motion map(RDMM)are obtained.Similar to the depth channel processing,RDMM is interacted with DBDMM to get RGB binary difference motion map(RBDMM).Then RBDMM inputs VGG19 convolution neural network to train and extract features.Finally,the features obtained from depth channel and RGB channel are fused.(3)A multi-feature fusion method: Fusing RGB and depth information into highly differentiated features can improve performance of action recognition.Many methods have proved that late feature fusion can improve recognition performance.In recent years,principal component analysis(PCA)has been widely used in signal processing,pattern recognition and digital image processing.PCA can minimize the loss of information,the main features are extracted and the linear correlation among multi-source data are removed.Information Entropy solves the problem of information quantification.Generally speaking,the smaller the information entropy of feature is,the larger the information content is.This feature is endowed with greater weight,which can play a more important role in comprehensive evaluation.Based on the advantages of PCA and information entropy,an improved feature fusion method based on information entropy improved PCA(IEPCA)is proposed in this paper.The calculation process of IEPCA is as follows: Firstly,the covariance matrix of features is constructed,and the eigenvalues and eigenvectors are calculated.Secondly,according to the eigenvalues,the image feature contribution rate and information entropy are calculated,and the corresponding weights are obtained.Finally,weighted feature fusion is performed.In this paper,the feature description and fusion of RGB image sequence and depth image sequence are deeply studied.The comparative experiments show that the proposed action recognition algorithm based on multi-feature depth motion map and RGB-D video can achieve better recognition results,and can be used for action recognition.