Research On Multimodal Hand Segmentation And Tracking

Hand segmentation and hand tracking have potential application prospects in the field of human-computer interaction and hand rehabilitation training.At the same time,the subject involves computer vision,image analysis,artificial intelligence,probability statistics and pattern recognition,having great theoretical research values.Most of the existing hand segmentation and hand tracking techniques use a single data modality.Each single modality has inherent defects,resulting in the technical performance cannot meet the practical requirements.For example,the depth image noise contains high noise and has low resolution.Although some other technologies adopt a variety of data modalities,there is still a huge room for improvement.Therefore,the thesis first studies the hand segmentation combining color image information and depth information,and then uses the segmented depth map and inertial sensor data as input to study the multimodal hand tracking technology.The specific research work of this thesis is as follows:(1)To solve the problem that depth images have low resolution and high noise and color images have high processing complexity,hand segmentation technique based on color image and depth image is proposed.Firstly,the thesis study how to suppress a class of unique noises of the depth map.Afterwards hand segmentation is carried out in the depth direction based on the depth threshold,eliminating most background pixels.Then,the original YUY2 code stream collected by the sensor is directly mapped into Y,U,V pixel values,and the second division is performed in the YCrCb color space,which is a guarantee for reducing the complexity of the proposed segmentation algorithm.Finally,because it is difficult for skin color segmentation to distinguish between the hand and the forearm,the arm region in the image is further removed based on the contour feature.Theoretical analysis and experimental results show that the proposed multimodal hand segmentation scheme has the advantages of low computational complexity and high segmentation accuracy.(2)Aiming at the low accuracy and the poor robustness to self-occlusion gestures and fast motion gestures of existing gesture tracking technology,a hand tracking technique based on depth image and inertial sensor data is proposed.Firstly,the 3D human hand kinematics model and the observation model are established.Afterwards a cost function corresponding to the model is designed to measure the difference between a pose hypothesis and the observation data.Then,the initial poses are generated by combining depth image and inertial sensor data from which the optimization starts.Finally,a comprehensive optimization scheme is proposed to minimize the cost function better.Theoretical analysis and experimental results show that the proposed multimodal hand tracking scheme not only improves the tracking accuracy,but also improves the robustness to occlusion gestures and fast motion gesture.