Real-Time 3D Tracking For Augmented Reality

Along with the explosive development of computing technology, the computer vision research has achieved sustained progress in the past decades, publicly distributed in surveillance, informa-tion retrieval, object recognition, navigation, medicine, education, etc. Mixed reality is an impor-tant application of computer vision, which is trying to exhibit the seamless mixture of real and virtual worlds, providing auxiliary information, such as description text, video tutorial,3D anima-tion, etc.Augmented reality (AR) is a research direction of mixed reality. It analyzes the object or scene features using related computer vision techniques, and augments the reality with the computer-generated information at the specified locations, helping people to understand the scene better. Generally speaking, AR system contains the following components, video input, feature analysis, camera pose estimation, and rendering, among which feature analysis and camera pose estimation are the cores of the system. Offline AR has been widely used in movies and TVs, but real-time AR is still in the experimental stage. The dissertation will focus on the 3D tracking technique of real-time AR, i.e. recovering the camera pose at real-time rate. The main topics include the parallel computing, the analysis of image features, the keyframe-based representation of large-scale scene, and the bi-layer segmentation with rotating camera. All in all, the dissertation aims to promote the use of real-time 3D tracking in augmented reality, and contributes as follows.●A unified real-time AR framework is presented. The whole AR system is split into several key modules, and the interfaces of modules are standardized to unify the AR systems for different environments in a parallel computing framework. The user can design new AR applications on the basis of the framework easily. The framework fully utilizes the computing power of multi-core CPU, and makes the AR systems always efficient and robust even in complex environments.●A refined fiducial marker system is proposed. It's inevitable to use artificial fiducial markers in the texture-less desktop environment. A Chinese character-based fiducial marker system is proposed to assist Chinese learning. In order to detect the fiducials under varying illumi-nations, the system utilizes the edge detection method to extract the bounding contours of markers. Further, the particular structure of Chinese character is captured by a distance field from the character contour to the bounding box, which improves the recognition. Mean-while, to alleviate the unfriendly black and white appearance of traditional fiducials, the natural images can be used as special markers.●A keyframe-based scene representation with a fast candidate keyframe recognition method is proposed. In large-scale natural scene, the structure-from-motion technique is used to recon-struct the 3D point cloud of the scene from the input sequence. The performance of feature matching drops quickly because of the abundant features in the large-scale scene. With the greedy optimization, a set of keyframe is selected from the input sequence, which contains as many stable feature points as possible. During real-time 3D tracking, the online image is matched to the candidate keyframes, which are determined by fast image recognition method. To achieve better tracking result, epipolar geometry constrained feature matching and temporal information are used to get more feature matches.●A bi-layer segmentation method with rotating camera is proposed. The scene layer segmen-tation method is important for handling occlusions in AR system, as well very difficult due to the complex movements of the scene and camera. A simple case of rotating camera before a static background is considered here, which is equally a bi-layer segmentation problem. The background panorama is constructed beforehand, then the online image is registered to the panorama and the corresponding background is recovered. The Graphcut is used to solve the segmentation. To handle the complex panorama and background registration error, a local color model combined with over-segmentation is proposed. With the help of the background contrast attenuation, the experiments show precise segmentation results. Some interesting augmented effects are also implemented.