Research On Dynamic Hand Gesture Recognition Based On Spatio-temporal Frature Learning Model

Hand Gesture recognition has been widely applied to human-computer interaction visual communication,virtual reality and other fields.More and more researchers pay attention to hand gesture recognition.Hand gesture recognition can be divided into static hand gesture recognition and dynamic hand gesture recognition.A static hand gesture refers to a still hand pose,taking into account only the spatial information.The dynamic hand gesture refers to a continuous hand gesture action,and contains not only the spatial information,but also its temporal movement information.Because a natural hand gesture is usually dynamic,dynamic hand gesture recognition has become a hot research issue because of its universality and convenience.However,dynamic hand gesture recognition is still a challenging problem.The complexity and variability of hand gestures and the environmental factors such as occlusion,illumination,viewpoint,individual differences all increase the difficulty of robust spatio-temporal representation and classification.One key issue in dynamic hand gesture recognition is the representation and extraction of hand gesture features.Traditional methods are usually based on predefined features,and the extracted hand gesture features are not very accurate and require considerable prior knowledge and manual adjustment.In recent years,the performance of the feature extraction method based on learning has been improved,and the extracted features have good robustness,so the learning-based methods have gradually become the most popular feature representation methods.Two classical models are convolution neural networks and Restricted Boltzmann Machine,but the traditional Restricted Boltzmann Machine are usually based on vector variables and matrix-based variables.Aiming at the dynamic hand gesture recognition application,this thesis improves the traditional Restricted Boltzmann Machine model,and proposes a 3D-2D Restricted Boltzmann Machine for 3-order dynamic hand gesture video data,that is,the input is a third-order tensor variable,while the output is the second order matrix variable,and we also propose a two-channel 3D-2D RBM to fuse spatial and temporal information We also propose a hybrid CNN-MVRBM-NN model.The improved model can effectively extract the temporal and spatial information contained in 3D video data.In the details are as follows:(1)Dynamic hand gesture recognition based on 3D-2D RBM model.First,we propose a 3D-2D RBM model.Furthermore,a two-channel 3D-2D RBM is proposed for dynamic hand gesture recognition application.Among them,one channel models hand gesture movement characteristics,and the other channel models spatial features,finally,the two channels are fused for decision.(2)Dynamic hand gesture recognition based on CNN-MVRBM-NN.This thesis proposes a dynamic hand gesture recognition method based on a hybrid model called CNN-MVRBM-NN.The model consists of three sub-models.The CNN sub-model automatically extracts the frame-level spatial characteristics of the dynamic hand gesture sequence;and then we organize the sequence features line by line to form a matrix,finally,the MVRBM sub-model models the spatio-temporal high-level semantic features of dynamic hand gesture.To increase the hand gesture classification accuracy,this thesis initializes the NN by the well-trained MVRBM,and then the NN pre-trained is fine-tuned by back propagation so as to be more discriminative.In this thesis,the proposed two methods are verified on the Cambridge hand gesture dataset,and the experimental results show the effectiveness and robustness of the methods.