Gesture Recognition Based On Deep Learning And Its Application In Virtual Experiment

In the field of gesture recognition,because of the complementarity in multimodal data,gesture recognition using multi-modal data can improve the recognition accuracy effectively.First of all,we propose a novel method for feature extraction of multimodal gesture data called modality-convolution.It not only extracts the intra-modality information,but also the inter-information from multi-modal gesture data.At the same time it complete the pixel-level data fusion so that the complementarity of information contained in multimodal data is fully utilized.Based on the modality-convolution,we describe a modality-CNN for multimodal gesture recognition,which convolve the original input image by modality-convolution.It can extract the spatial feature,temporal feature and inter-information feature at the same time,and use them to classify the gestures.Secondly,we build a gesture recognition model based on the modality-CNN to complete the dynamic gesture recognition.It consists of 4 parts:preprocessing,feature extraction,data fusion and gesture classification.Among them,the part of preprocessing complete the data augmentation,cutting and so on.The part of feature extraction adopt the modality-CNN and DBN to extract the features of gesture video data and skeleton data separately.The data fusion part realizes the decision-level fusion of two kind of features.The part of gesture recognition uses HMM to complete the recognition of dynamic gestures.Finally,we use leap motion to collect gesture data,and develop a computer disassembly virtual experiment based on gesture recognition technology.It can identify user gestures effectively,provide more natural human-computer interaction to user to finish the virtual experiment.We finish a comparative experiment on Cha Learn LAP 2014 gesture dataset which use Jaccard index to calculate the validation accuracy test accuracy of the modality-CNN the whole gesture recognition model.The result shows that the modalityCNN is more accurate than the baseline by 3%,and the gesture recognition model is more accurate than the baseline by 2.3%?So the modality-convolution is able to extract the interand intra-modality information effectively,which is helpful to improve the accuracy of multimodal gesture recognition.