Hand Gesture Recognition Algorithm Based On Multimodal Input

As the vanguard of a new wave of science and technology,artificial intelligence is penetrating into every aspect of human life at an unprecedented speed.Human-computer interaction(HCI),as an improtant part of artificial intelligence,has been re-ceiving widespread attention.Among the numerous means of human-computer inter-action,gestures recognition technique,a natural HCI method which is closest to human communication habits,has been widely used in deaf-mutes education,intelligent house system,as well as virtual reality and so on.In the above context,both vision-based static and dynamic hand gesture recognition are carried on the thorough research in this dissertation.The main works and contributions of this thesis include the following aspects:1.Static hand gesture recognition is studied deeply.Traditional gesture detection methods can not distinguish the forearm,the palm and the fingers well,which has great impact on gesture recognition.To handle this problem,we propose a markless lined-based forearm removal algorithm to search the best hand cropping position,i.e.,the position of wrist,which can remove forearm exactly to get the hand region.2.Most of existing static gestures recognition algorithms use shape decomposi-tion method to extract fingers,then template matching technique is adopted to classify gestures.Therefore,the performance of the finger detection algorithms has a great in-fluence on the robustness and accuracy of the whole system,so this dissertation makes three improvements:(1)A new gesture shape decomposition approach which is based on morphological processing and curvature analysis,is proposed to extract finger re-gions accurately.(2)This paper proposed an improved multi-parameter similarity mea-sure method.(3)A hierarchical template matching recognition method is presented in this dissertation.Experimental results show that the proposed system can obtain high recognition performance,and can overcome the influence of scrambled background,approximate skin color regions and other adverse factors effectively.3.An efficient multi-model fusion method based on convolutional neural network is proposed to classify isolated dynamic gestures.Given a depth map sequence,the pro-posed method extracts the motion information at different levels firstly,and then feeds it into multiple convolutional neural networks of different structures to predict the as-sociated three-dimensional temporal information.Based on this,we can capture the continuous motion characteristics of hand gestures from the spatial and time space,and thus achieve the classification of dynamic hand gestures.Both qualitative and quanti-tative results demonstrate the effectiveness of the proposed method.