Research On 3D Hand Motion Tracking Based On Depth Images

Analyzing hand motion by means of computer vision is an important research area, which includes hand detection, hand pose estimation and action recognition. This area is multidisciplinary, involving image processing, computer vision, computer graphics, pattern recognition and artificial intelligence etc.. Vision analysis of hand motion has various applications, including virtual reality,3D animation, robot learning by imitation, advanced human-computer interaction, sports biomechanics etc..This thesis focuses on hand pose estimation, namely, estimating hand global position and orientation and each finger joint angle from visual observation. Hand pose estimation is the core problem of hand motion analysis, and it’s the basis of action recognition. In this thesis, hand pose estimation is aimed at an image sequence. On the assumption that the hand motion is coherent, state transfer model is introduced, and single frame pose estimation is turned into a motion tracking problem on a sequence.Articulated hand motion tracking via computer vision is very challenging, due to a number of complicating factors. The high-dimensionality of the hand state space brings difficulties to the search of the global optimal, leading to a large amount of computation. The self-occlusions that occur frequently in hand motion, will cause the ambiguities of the observations and the multi-peak probability distribution of the hand state, making it more difficult to search for the global optimal. The dynamics of hand motion and the observation likelihood are both non-linear, requiring that the tracking algorithm have the ability to solve nonlinear problems.An effective hand motion tracking solution must tackle the following key problems:(1) Constructing a good matching error function. The matching error function is used to describe the incompatibility between the hand pose hypothesis and the observations. The construction of the matching error function is the foundation of hand motion tracking, which determines how difficult to search for the global optimal during the tracking process. A good matching error function will reduce the local minima around the global optimum, making it easier to search for the global optimal. (2) Developing an effective search method. The high-dimensional state space and the multi-peak probability distribution demand a strong search method for the global optimal. The search method must have a good convergence rate and an ablity to jump out of local minima.A lot of research has been done to deal with the above problems. However, it’s still difficult to simultaneously meet the requirements of speed, accuracy and robustness for practical applications. In this thesis, using depth images as input, based on the improved particle filtering algorithms, the 3D marker-free hand motion tracking is studied. Specifically, the main work that has been done in this thesis is as follows:(1) The hand model and the observation model for 3D hand motion tracking are established. In this thesis, a 26 DOF hand kinematics model with anatomical constraint is used. To balance model accuracy against computational complexity, hand shape model is built with basic geometric primitives. Then, using depth observation from an Kinect sensor as the system input, a matching error function is constructed with depth features and silhouette features, to measure the difference between the hand pose hypothesis and the observations.(2) To deal with the high-dimensional sampling difficulties of particle filtering, two improved particle filtering algorithms are proposed for 3D hand tracking, based on the thought of integrating swarm intelligence optimization into particle filtering to improve its sample distribution. The first algorithm combines differential evolution with particle filtering, using differential evolution to move the particles to the area with a higher likelihood. The second algorithm employs particle swarm optimization particle filter algorithm for hand motion tracking, and to prevent premature convergence, simulated annealing and partial randomization are both used to improve the algorithm. Experiments show that, both of the algorithms can track 3D hand motion effectively and robustly, but the second one is slightly better than the first one in accuracy.(3) By establishing models both for the hand and the object, the tracking of the hand interaction with an object is studied. In the real world, the action of the human hand is usually interactive, and the interaction with objects is the most common case. The presence of the object increases the complexity of hand motion analysis. On the other hand, the useful information carried by the object context will help in recognition and estimation of the hand motion. This thesis uses the model-based method for tracking the hand interaction with the object. 3D shape models and dynamics models are established both for the hand and the object. During the tracking process, both the hand and the object are tracked. With single depth images as input, using improved particle swarm optimization particle filter as the tracking algorithm, the proposed method can effectively track the hand interaction with the object.(4) According to the characteristics of the proposed tracking algorithms, based on the multi-thread rendering engine OSG and off-screen rendering techniques, two kinds of 3D hand tracking prototype system are developed:the single virtual camera system and multiple virtual camera system. In the framework of particle filtering, single virtual camera system just creates one virtual camera for matching error calculation, and for each OSG frame, only one particle is rendered and calculated. However, multiple virtual camera system creates virtual cameras for an entire generation of particles, and for each OSG frame, all the particles of the generation are rendered and calculated through corresponding virtual cameras.