Research On Real-time Hand Gesture Recognition Based On RGBD Depth Image

Hand gesture recognition is a research hotspot of computer vision and human-computer interaction areas. It has broad application prospects in many fields, such as virtual reality, robot remote welding, intelligent driving, office assistance, entertainment and sign language recognition. With the development of the software and hardware of the computer, depth image captured by depth camera has been widely used in hand gesture recognition because the three-dimensional information of the depth image could provide more useful data.In this paper, the Kinect depth camera is used to capture RGBD depth image to deal with the hand gesture in real time, including the hand gesture segmentation and recognition of static hand gestures and dynamic hand gestures. On the basis of the research, the prototype system which embodies the driver-computer interactive simulation platform based on real-time hand gesture recognition is implemented. The main work of this paper is summarized as follows:1) To segment hand from the complicated background, a method of hand gesture segmentation based on motion, dynamic threshold and skin color(MTS) is proposed. Consisting of locating motion area by motion information, extracting hand area by dynamic threshold and positioning skin-color region by skin color mapping to HSV and YCbCr color spaces, this method could be used in real-time hand gesture segmentation effectively.2) To extract significant features in static hand gesture recognition, the multi-feature fusion method which means the hand’s contour features and geometrical parameter features are combined with Hu invariant moment features is presented. Then the static hand gesture dataset SHGD is made as the input of SVM classifier for training. Utilizing the training model, the experiment shows that the method can recognize specific 8 static hand gestures designed with good accuracy.3) In the process of dynamic hand gesture recognition, to match the gestures with effect, the matching algorithm dynamic time warping(DTW) is mixed with the classification algorithm kNN, the new algorithm called k-DTW. By collecting templates of specific dynamic hand gestures, the dataset of dynamic hand gesture(DHGD) is made. As the computation phase of kNN, DTW could calculate the distance effectively. As well as the distance can be used as the weight in the correlation calculation for voting on classification. Using k-DTW method to deal with the recognition of 4 dynamic gestures designed performs stably with good accuracy in the experiment.4) Based on the research of static and dynamic hand gesture recognition, the recognition is applied to the development of the prototype system. This paper designs a driver-computer interactive simulation platform based on real-time hand gesture recognition. For better understanding of the principle, structure and function of the simulation platform, a vehicle-mounted hand gesture recognition interactive prototype system is accomplished. The prototype system implements 3 modules, including telephone answering module, music playing module and lane departure detection and warning module. The system, which could help the drivers while driving in the simulation environment, runs well with good rate of recognition, instantaneity and robustness.