Multi-Fingertips Tracking In3D Space Using Kinect And Its Application

Real time detection and tracking of3D multi-fingertips can make users feel as if operating real objects when they are actually opreating virtual objects, so bringing good experience to users. It can also provide3D structural information of the hand, so making gesture recognition more accurate and robust.In this paper, we introduce a fast and robust algorithm for tracking3D multi-fingertips using Kinect. Our method firstly uses Kinect to get a rough hand region following by applying Bayes model for it to get more correct hand region, then applies fingertip detection method based on pixel classification to get2D fingertips and their directions. Finally, we sample on the depth map around the detected fingertips to get the Z-value of the fingertips and then combine the Kalman Filter and the continuity between frames for the tracking of3D fingertips. The fingertip detection method which is come up with according to the specialty of the data of Kinect and the Kalman filter applied with continuity between frames are the key of our algorithm and also our two contributions.For the fingertip detection method and3D fingertip trajectory tracking method we come up with, we separately implement one2D HCI application called "Telekinesis Touch" and two3D applications called "Put Block Object Away" and "Browse virtual object". And we also do some experimental and user study for these three applications. The experimental results show that our2D fingertip detection method can correctly detect the positions of the fingertips even if in the situation of motion blur and finger bent, and our3D fingertip trajectory tracking method can robustly track the fingertip’s3D trajectory, so making the3D HCI application implemented with it brings good experience to users.