Research On Fingertip-Based Gesture Recognition And Human-Computer Interaction Applications

With the development of the computer technology, interactions between humans and computer are becoming increasingly common and diversified. Now the main types of human-computer interactions are based on the mouse, keyboard and touch screen. Although these interactions have been very mature and can be a very good form of interactions between humans and machines, they have certain limitations and can not release the operation ability of hands. So, more and more researchers began to study non-contact human-computer interactions which can identify gesture and realize interactions by using the capture camera. This form of interactions frees hands from the restraint of the device and gives full play to potential of hands. The Kinect device provided by Microsoft in 2010 can obtain color data, depth data as well as human skeleton data, and it can make the human-computer interaction based on machine vision more convenient. In this thesis, based on Kinect and the fingertip information, static gestures and dynamic gestures are recognized and encoded as instructions to control software or hardware applications, so non-contact interactions between humans and machines can be realized.In this thesis several commonly used static gestures and dynamic gestures are taken as the basic recognition target. For static gesture recognition, the gesture in the color image area is initially determined rapidly by using the depth information and skeleton information of Kinect, and the hand posture region is extracted from the gesture area by using the YCrCb skin color model. Then the palm point and fingertips point can be acquired by using the static gesture palm point algorithm and fingertip point algorithm proposed in this thesis. Then the commonly used static gestures are identified by combining bent finger point features, the ratio threshold of distances and the maximum amount of contour area. For dynamic gesture recognition, through the palm point algorithm and fingertip point extraction algorithm proposed in thesis, the finger point trajectory can be tracked and recorded. Then dynamic gestures are recognized with the help of the trend of the trajectory, mutation point features,, the angle threshold and the distance ratio threshold.The static and dynamic gesture recognition methods based on Kinect and the fingertip information proposed in this thesis can effectively recognize commonly used human gestures. By using the depth data to narrow the gesture area in the stage of gesture segmentation based on YCrCb skin model, it can appropriately expand the threshold range, and the influence of illumination on human bodies can be greatly reduced, therefore the stability of gesture recognition can be improved. By integrating the recognized gestures as instructions, humans can interact contactlessly with the machine car, the PPT demonstration and the characters in the game, and it can have good application value in the future.