Dynamic Hand Gesture Recognition Based On Deep Convolutional Neural Networks

As one of the great inventions of the 20th century,computer not only enhances the efficiency of human work but also enriches people's daily life.It has become an indispensable tool for human life.Therefore,how to interact with computers with high efficiency,naturalness and comfort has become a research hotspot of researchers and has produced a new subject,Human Computer Interaction(HCI).At present,the use of visual information to interact through artificial intelligence has become mainstream in recent years.The most used information in vision-based human-computer interaction is faces,limbs,and gestures,and gestures are widely used in various intelligent interactive products due to their natural,rich,and comfortable performance characteristics.The gesture recognition algorithm is divided into static gesture recognition that recognizes the hand shape in the space and dynamic gesture recognition that recognizes the gesture transformation in the image sequence.Dynamic gestures have more uncertainty in space and time,making identification more difficult.The research of visual gesture-based dynamic gesture recognition is divided into two categories:one is the traditional dynamic gesture recognition method,which includes four stages:gesture segmentation,gesture tracking,gesture feature extraction and gesture classification.Each stage has very strict requirements on the algorithm.Strict;the other is dynamic gesture recognition based on convolutional neural network.This method can directly obtain the classification result by sending the image sequence into the designed network structure,and the recognition process is simple and accurate.In this context,this paper chooses to study dynamic gestures based on convolutional neural networks.The main work is as follows:(1)The traditional vision-based dynamic gesture recognition is studied.The steps and methods are summarized.(2)Multimodal joint training method based on 3D convolutional neural network is proposed.Firstly,the 3D convolutional neural network is used to train the RGB image.Then,the Depth image and the edge image are trained based on the RGB training model.Finally,the recognition results of the three are combined to improve the accuracy of dynamic gesture recognition.(3)We proposed multi-direction 3D convolutional neural networks.The 3D convolution is applied from three directions of the video block to extract the spatio-temporal features of the dynamic gesture,and then the spatio-temporal features of the three directions are merged in different ways to achieve feature complementation.We compared the max fusion,concatenation fusion and multiplication fusion,the concatenation fusion get the best accuracy of 77.1%under three modal multiplication fusion.(4)We use the optical to extract the key frames from origin video.Compared with the uniform sampling,it can effectively extract frames with important motion information to improve recognition accuracy.In order to resist the overfitting of the convolutional neural network,a video data augmentation method is proposed.