Deep Learning And Path Signature Based Gesture Recognition

In recent years,gesture interaction is attracting more and more attention.At the same time,with the applications of depth cameras like Kinect,Real Sense and the development of pose estimation algorithms,we can acquire key points coordinates of human body from image or video easily.Although a significant progress has been made in key point trajectory based gesture recognition,it still faces some challenges:(1)less input information;(2)scale variation;(3)intra class differences;(4)inter class similarity.The existing methods can be mainly divided into feature based methods and deep neural network based methods.Feature based methods often lacks generalization and requires some domain knowledge.Deep neural network based methods perform well but need large calculation and careful hyper parameter adjustment.Based on the above point of view,this article makes an in-depth study and exploration of key points based gesture recognition from aspects of feature designing,network designing and the combination of feature extraction and network training.The research includes: 1)gesture recognition based on hand trajectory and its path signature feature.2)gesture recognition based on skeleton information and its path signature feature.3)gesture recognition based on path signature convolution neural network.The key part of single key point trajectory based gesture recognition is the modelling of temporal information.We propose two network architectures named PSNet(Path Signature Network)and LPSNet(Log Path Signature Network),which use the path signature feature to model the hand trajectory.Our networks can not only model the temporal information,but also combine the color and depth videos at the feature level.Moreover,we also propose a novel data augmentation method called Drop Frame to ensure the data diversity and model robustness.The experimental results on SKIG dataset prove the validity of our algorithm.As for multi key points based gesture recognition,beside temporal information modelling,the key parts also include spatial and spatiotemporal information modelling.We define two kinds of temporal paths and one kind of spatial path and further extract three kinds of features named temporal path signature feature,spatial path signature feature and spatiotemporal path signature feature.In the respect of network architecture,we propose a temporal transformer module(TTM)to align the key frames among different skeleton sequences.We use a multi-stream fully connected network to do classification.We prove the performance of our algorithm on Chalearn 2016,Chalearn 2013 and MSRC-12 three datasets.The above works have two limitations: 1)feature extraction and network training are independent of each other.2)the paths need to be defined beforehand.To solve these problems,we propose a path signature module(PSM),and further design three types of PSM: deformable path signature module(DPSM),temporal path signature module(TPSM)and spatial path signature module(SPSM).Finally,we propose an end-to-end path signature convolution neural network(PSCNN)based on PSM.We verify our network on Chalearn 2013 datasets.In summary,we study trajectory based gesture recognition from feature extraction,network designing,the combination of feature extraction and training process three aspects.In the aspect of feature extraction,we leverage the PS feature and design several types of PS features to model the spatio-temporal information.In the aspect of network designing,we propose TTM to align key frames and simplify the network architecture by robust feature designing and efficient feature fusion.Finally,we propose the DPSM to combine the process of feature extraction and model training,it can learn the paths automatically and avoid the weakness of predefined paths.