A Study On Kinect-Based Hand Gesture Recognition And Image Stitching For Virtual Reality

Virtual Reality(VR)has become very popular nowadays,and it has a significant impact on the way man-machine interaction and immersive experience.In these technologies,action recognition and panoramic image display are two very important research contents.The core topic of human-computer interaction(HCI)is vision-based interface(VBI),keyboard and mouse as a mechanical input device,in some ways,it is difficult to 3D and high input of freedom.Hand gesture in man-machine interaction,with vivid and intuitive features,has a strong visual effect and can be used as an interactive means.However,before hand ges-ture recognition task,the detection and positioning of hands is also a priority.Therefore,hand segmentation as more basic work as well as hand gesture recognition are studied.In another research,panoramic image display is the basic research content in VR technology.At present,image stitching algorithm is mainly used to synthesize the panoramic image by images with overlapping areas obtained from multiple angles by ordinary camera.Thus,our goal is to achieve seamless mosaic image accurately and make the mosaic image look comfortable.In the thesis,we focus on automatic hand segmentation and hand gesture recognition tasks based on Kinect camera and local homography-based image stitching re-search.Based on the previous methods and related studies,some solutions and innovations have been proposed.The main work of this thesis is as follows:1.An interaction-free hand segmentation algorithm is proposed.To extract hand segments using Kinect depth camera,user interaction-based semi-automatic approaches have been popularly studied.However,their segmentation performance highly depends on user's input.Moreover,it is redundant for users to undergo trial and error which finds good user's input for sophisticated hand segmentation results.So,we obtain the seed for hand segmentation without any user interaction on the depth map using a variable-scale circle.This circle size is varied for the hand shapes from different persons.Then,we refine the hand segmentation result around the seed point on the color image based on Gibbs random field.2.A gesture recognition algorithm based on SPFEMD distance measurement is proposed.Fingers represent informative hand gestures,and SPFEMD measures dissimilarity between fingers and hand gestures.SPFEMD is not only robust to deformation and distortion,but also invariant to scale translation and rotation.We use color and depth information captured by Kinect camera to perform automatic hand segmentation without any user interaction.Then,we extract fingers from the segmented hand using a morphological operation.Next,superpixel finger earth movers distance(SPFEMD)as a distance metric,is proposed to mea-sure the dissimilarity between the hand gestures based on template matching technique for represented fingers part.3.An image stitching algorithm with matching distribution constraints is proposed.It applies a smooth stitching areas throughout the entire target image and computes all the local transformation variations.For the feature matching step,we constrain the extract-ed SIFT feature to ensure the uniformity of distribution in the local matching region and not be disturbed by local extreme point for calculating global similarity transformations.Final,by comparing the global similarity transformations with the proper projection,the non-overlapping region can be adjusted.