Study Of Issues Of Geometric Consistency On Augmented Reality

Augmented Reality (AR) is a kind of mixed reality technology. Video-based AR extends a user’s visual perception by embedding virtual objects or additional information into video streams of the real environments in real time. In order to achieve photorealistic mixture effectives, it is necessary for the virtual objects to keep consistent with the real environment in geometrical and lighting conditions, among which the geometric consistency is the most basic constraint for augmented reality, which requires virtual objects to be rendered with right locations, perspective projection parameters and occlusion relations. The complete geometric consistency topic in augmented reality includes several sub-problems, such as3D reconstruction, camera tracking and occlusion between real and virtual objects. Different sub-problems are often related with each other. For example, camera tracking algorithms often depend on certain known geometrical structures of the scene, and geometrical information of the scene can also be updated while the camera is tracking. Thus, an effective geometric consistency strategy is usually a combination of several strategies on these sub-problems.Failure of geometric consistency typically leads to jitter or drift of the virtual objects, which can easily be found by users. Thus, robustness is an important criterion for a geometric consistency algorithm. Meanwhile, an AR system is also a real-time system, in which operations like video processing, parameter estimation and virtual object rendering should be finished online. System delay is therefore another important criterion for a geometric consistency algorithm. Traditional AR systems employ simplified models to describe the real environments, which are aimed to reduce computational complexity and keep the system running in real time. But using simplified model weakens the ability of the system to deal with complex environment information. Using more sophisticated models enables us to overcome these disadvantages, but also brings higher system delay. Therefore, a high-performance geometric consistency algorithm should keep a good balance between robustness, flexibility and system delay.This thesis provides a comprehensive survey on state-of-the-art works on several sub-problems of the geometric consistency topic, including feature extracting and matching, real-time camera tracking and depth map enhancement, and proposes new solutions to these problems. The innovation and contribution of this thesis mainly include:1. Proposing a real-time feature tracking algorithms. Feature points can be extracted and tracked among image sequences, and the tracking result can be used for3D reconstruction and real-time camera tracking. But a common feature tracking algorithm involves a number of image processing steps and floating-point computation, which is quite computationally expensive. We propose to use general-purpose computing on GPU technology to accelerate KLT algorithm, which achieves a200-300times speedup and largely reduces the system delay.2. Proposing a real-time camera tracking algorithm based on hybrid feature points. Compared with earlier camera tracking algorithms, which relay on markers or reference objects, a feature-point-based camera tracking algorithm is more flexible and has less limitation on the scene. But different feature tracking algorithms vary greatly on their tracking abilities and time delays. We compared SIFT and KLT feature tracking algorithms, and propose a real-time camera tracking algorithm based on SIFT and KLT hybrid feature tracking, reducing the time-cost of camera tracking among consective frames to no more than15ms, which maintains a good balance between robustness and system delay.3. Proposing a depth map enhancement method based color and depth consistency. The development of low-cost depth sensor, such as Kinect, gives us a convenient way of capturing depth data. But depth maps generated by these low-cost depth sensors are often found to be noisy and even contain many large holes. We proposed a novel depth map enhancement method based on color and depth consistency. By fusing raw depth values with image color, edges and smooth priors in an optimization framework, both misalignment and large holes can be eliminated effectively.4. Proposing and implementing a visual tele-AR system. This prototype system is a combination of tele-existence and augmented reality technology, which is built based on several new algorithms formerly proposed in this thesis. It is very robust and runs in real time, which also proves the effectiveness of our former works.