Hand Gesture Interaction Technique Research And Application Based On RGB-Depth Cameras

Hand gesture interaction is a very important research topic in human-computerinteraction, and it has gained a lot of concerns and applications in the field of virtualreality, robot control and interactive games in recent years. As a natural and friendlyinteraction way, vision-based gesture interaction is a significant branch of gestureinteraction, and it is also the current research focus and future development trends ofhand gesture interaction. In this paper, we study the key technologies of vision-basedgesture interaction with the use of RGB-Depth cameras, the main work andachievements are as following:(1) A hand detection approach is proposed based on skin and depth informationby the use of RGB-Depth cameras. Skin segmentation and shape-based methods areusually used in traditional hand detection, but it is difficult to get satisfyingsegmentation hand area under complex background. To solve the problem, wecalibrate the RGB camera and depth cameras at first and obtain the projected imageby projecting the depth image to RGB coordinate system. Then hand detection whichcombines the depth segmentation method and an improved skin segmentationmethod is done using the projection image. The experimental results demonstrate therobustness of the novel approach, when in the case of hand overlapped with user’sface or other person in the background, we can still get the hand image in goodcondition.(2) A gesture recognition approach combing the depth image is proposed by theuse of RGB-Depth cameras. From the the perspective of classifier fusion, the gestureclassifier is trained using AdaBoost algorithm and depth image, then the Haar-likefeatures are extracted with the segmented hand image and finally specific gesturesare recognized. The experiments reveal that the novel approach runs fast and canmeet the real-time requirment. Besides, the recognition rates are improved to acertain extent, and the comprehensive performance of the proposed approach israther good.(3) A gesture interaction system is conducted based on the RGB-Depth cameras.First the color images and depth images are captured using the RGB-Depth cameras,then we use the proposed hand detection and gesture recogntion methods torecognize the gesture alphabets in American Sign Language, finally the system performance is evaluated from the processing speed and application scope, and theexperiments show the effectiveness of the hand gesture interaction system.