Gesture Based Natural Interaction In Augmented Reality

One of the important goals of Augmented Reality (AR) is to achieve direct and natural interaction between users and virtual information. Under the framework of AR, our work extends the concept of 3D registration. We present a method to register the 3D space and 3D shape of foreground in the real world. In addition, we reconstruct the 3D sparse-structure of markerless hand to get interaction informations. We achieve our goal by creating a 3D interaction method in which users can use bare hand to interact with virtual computer graphics, with correct occlusion relationship presented. The main contributions include the following:(1) Present an efficient algorithmic solution to resolve the occlusion problem of bare hand real-time interaction in AR environments. We propose a stereo vision based method to handle the occlusion problem in the view of a potentially moving camera. We carve the visual hull and extract the "depth-texture" of the real object. Hence we use this information to render the correct occlusion relationship between real and virtual objects.(2) Give the analysis and classification of "extrinsic matrix registration", "extensible tracking", and "3D space registration". We extend the concept of 3D registration and classify the space consistency problem into the area of 3D registration of AR.(3) Present a series of methods about marker-free hand detection; hand feature extraction and hand 3D structure reconstruction. Based on this, we extract 3 basic 3D interaction structures of hand:Pont, Line and Plane in real world and align them in the virtual world to achieve space unified hand based natural behavior 3D interaction. We also utilize a multi-cue gesture tracking method to achieve multi-point tracking imitation gesture interaction applications.(4) Develop a system applying our algorithms to embody and verify our concept. We implement several applications based on this system, which can give rapid develop support of follow-up study. And we do not need complicate or expensive equipments; just need desktop PC equipped with common webcams.