Multi-Modal Based Hand Tracking

As an important research topic in the area of human-computer interaction(HCI),hand tracking has a wide range of applications,like animation,rehabilitation,robotics,virtual reality(VR),and augmented reality(AR).Most of existing solutions are based on single-modal inputs and the tracking object is a human hand that does not interact with external objects.Its performance is limited by the inherent shortcomings of these single-modal inputs and it is insufficient to deal with human-object interaction.Based on the modalities of inputs,this thesis studies the hand tracking algorithms using multi-modal inputs for human hands in non-interaction scenarios and human hands during manipulation respectively.The details are as follows:To deal with the problem that the existing works are limited by single-modal inputs,this thesis proposes an algorithm for non-interaction scenarios to tracking hands using a data glove and a depth camera.The data glove is used to fill the data gap that occurs with depth data during occlusions.And the depth data provides an absolute measurement of hand state,which helps reduce the dependence of data glove on tedious calibration to ensure accuracy.In order to combine data glove and depth data,this thesis presents several novelties on DoFs(Degree of Freedom)setting,hand shape adjustment,self-interaction,tracking corresponding computation,and pose searching space constriction to formulate the hand tracking as an optimization problem on hand pose parameters to suit the depth images.Experiments show that our algorithm can run in real-time with CPU and can effectively improve accuracy and robustness.To deal with the problem that existing solutions are sufficient to handle human-object interaction,this thesis presents an interactive hand tracking algorithm based on the state of interacted objects using a data glove and a RGB-D camera.We hold the idea that the state of interacted objects helps to infer interactions between hands and objects.Thus,this thesis designs a series of states for the objects after a deep analysis of the interaction between hands and objects.To consider the movement of hand and object and the interaction jointly,we formulate the interactive hand tracking as an iterative optimization problem with innovations in object tracking and prediction,hand-object contact,interaction and collision Experiments show that our algorithm can effectively improve accuracy and robustness.