Hand Gesture Recognition Based On Depth Information With Kinect Sensor

Since hand gesture recognition has been widely used in virtual reality, computer games, and sign language, it becomes a popular topic in human-computer interaction in recent years. Traditionally, there are two types of hand gesture recognition methods in terms of image capture devices. The first type is based on the data glove. It is inconvenient to use, and may hinder natural joint of the gesture, therefore, it is not a very popular method. Another type is based on the optical sensor. However, it is sensitive to lighting conditions and backgrounds, which restricts its applications in the real world.Thanks to the emergence of Kinect depth sensor, new opportunities are provided for vision-based hand gesture recognition. It can detect and segment the hand robustly using Kinect sensor. Despite the recent success in many ways for human tracking and human action recognition based on Kinect, Kinect-based hand gesture recognition is still an open question, and this technology used in human-computer interaction also has great potential. This dissertation studies the static and dynamic hand gesture recognition technology using depth information captured by Kinect sensor. The main works and contributions of this dissertation are provided as follows:1?A part-based static hand gesture recognition algorithm are proposed. Existing part-based hand gesture recognition methods with Kinect sensor segment hand shape using depth information, detect fingers using shape decomposition method, and match finger parts with templates using Finger-Earth Mover's Distance(FEMD). Because of FEMD's sensitivity to the finger part variations, this dissertation propose two enhancements to these systems. Firstly, this dissertation presents an improved FEMD to resist the influence of finger part variations and noise. Secondly, this dissertation presents a new discrete contour evolution-based hand shape decomposition method to segment finger parts from palm accurately and efficiently. When these two enhancements are embedded, both qualitative and quantitative results confirm that the overall part-based hand gesture recognition method can achieve state-of-the-art performance.2?A depth projection map-based static hand gesture recognition method is proposed. The depth map provided by Kinect is mainly used to segment hand shapes in many previous work, and the informative shape is not sufficiently utilized. This dissertation designs a novel effective descriptor to take full advantage of hand shape information from depth maps, and proposes a new static hand gesture recognition method based on this novel descriptor. Firstly, depth maps are projected onto three orthogonal planes to generate the depth projection maps. Secondly, Bag of Contour Fragments(BCF) descriptors are extracted from the three depth projection maps and concatenate them as final shape representation of the original depth data. Finally, gesture classification is performed. Experiments show that this method is of high recognition accuracy, and can overcome the impact of cluttered background, self-occlusion and other unfavorable factors effectively.3?A depth motion map-based dynamic hand gesture recognition method is proposed. Considering that the depth sequence of dynamic hand gesture carry a wealth of shape and motion information, this dissertation designs a novel effective descriptor to represent the gesture, and proposes a new dynamic hand gesture recognition method based on this novel descriptor. Firstly, each frame of the depth sequence is projected onto three orthogonal planes. Secondly, the binary map of motion energy is obtained by computing and thresholding the difference between consecutive maps, and the depth motion map is generated by stacking the motion energy through entire video sequences. Finally, gesture classification is performed. Experiments validate the effectiveness of the depth sequence-based dynamic hand gesture recognition method.