Research On Somatosensory Interaction Technology In Virtual Environment Based On Kinect Gesture Recognition

Nowadays, Human-Computer Interaction(HCI) applications in virtual environment mostly have been based on mouse, keyboard, digital glove and controller, which is difficult for users to get real and natural interaction experimence and seriously restricted the practical applications of the virtual environment. With the combination of somatosensory and virtual environment technologies, a completely realistic intelligent virtual environment can be generated, which consists of multiple senses including vision, audio, touching, feeling and emotion. More abundant and accurate information and knowledge also can be freely obtained from this digital virtual environment.According to the application demand of nature HCI in virtual environment, combining the somatosensory and virtual environment technologies, the thesis presents a somatosensory interaction approach based on hand gesture recognition.In this thesis, a virtual environment interaction method was developed based on gesture recognition. We take use of Kinect sensors to get gesture data and designed different algorithms to identify static and dynamic gestures respectively. In addition, according to daily habit of human gesture interaction, a virtual mine interaction system is designed based on gesture recognition, which realize the more natural human-computer interaction and can increase the user's natural and realistic experience. The main content in this thesis are as follows.Firstly, a natural interaction design method in virtual environment based on gesture recognition was proposed. On top of this, the natural interaction design of virtual environment, the key technologies of depth information acquisition and human skeleton tracking was studied, etc.Secondly, the research on the approach of static hand gesture recognition is captured by the means of Kinect sensor which utilizes the depth information and the gray histogram to extract gestures and split the hand palms up through pixel features that have been smoothed with median filter. Then combined with K-curvature and Convex Defects a method is demonstrated to recognize fingers. Finally, the experiments are conducted to test and evaluate the accuracy and robustness in different environments.Thirdly, when it comes to dynamic gesture recognition method based on Kinect, human joints information is extracted for tracking the dynamic trajectory of the hands. Furthermore, an improved dynamic time warping(IDTW) gesture recognition algorithm was proposed, which constrained the search path of traditional dynamic time warping(DTW) algorithm and weighted features. The testing results on public gesture data sets show that the proposed algorithm reduced calculation amount and also improved the overall recognition rate.Finally, a somatosensory interaction system in Virtual Mine based on Kinect gesture recognition is designed for the demand of natural human-computer interaction applications based on Virtual Mine. This system defines the Virtual Mine interactive gesture. In addition, the accuracy and effectiveness can be achieved by testing and verifying the proposed method in a typical interactive scenario.