Monocular Camera Based Perception In Augmented Reality Scene

By analyzing scene features, geometric information generated by computer can be superimposed on the real environment utilizing augmented reality techniques, and such a visual fusion can enhance human’s perception. As a new type of human computer interaction, the development of augmented reality technology has received much attention. Fusion between the virtual and real world is the key to augmented reality. Three technological challenges for creating an immerging augmented reality include:how to recover real 3D scene from 2D visual information; how to estimate the position and pose of the visual sensor in the space, and how to identify and perceive the objects existing in the scene. Despite the significant progress of computer vision, however, research challenges remain related to sensor noise, changing illumination, occlusion, identification and pose estimation of objects in clutter environment. Therefore, developing stable and efficient algorithm for scene reconstruction and perception is still a very challenging task.For these reasons, this thesis mainly focuses on building augmented reality assembly system, studying how to recover the 3D information and estimate the camera pose from the real captured video, which will be reused in the assembly instruction system. The difficulties in accurate registration, real time responding and the robustness in the practical situation will be effectively addressed which will accelerate the development of scene perception in Augmented Reality application. The main works of this thesis are summarized as follows:In order to solve the problems of low efficiency, difficulty of identifying and perceiving objects in the 3D reconstruction of a scene, a reconstruction algorithm is proposed which will recover a sparse point surface from sequences of images. The algorithm employs local bundle adjustment which will maintain the robustness and accuracy of the reconstruction, while improving the computing efficiency. Considering the noises inducing from vision measurement, a statistical outlier removal method is introduced. The method can effectively filter large amount of outliers without increasing the cost of computation, and the number of points will be reduced greatly while create a better reconstruction of the surface of the scene.In order to present the visual immersion of alignment and registration, an efficient camera tracking algorithm based on sparse optical flow is introduced to determine the pose of a moving camera in an Augmented Reality application. Taking into account of the instability, low efficiency in estimating the pose of a moving camera in an unprepared environment, a keyframes bundle tracking method is introduced. The method utilizes sparse optical flow to track features across adjacent frames of keyframes bundle, and relationship between correspondence features is established. The moving camera’s pose is updated according to the 3D location of these feature in scene point cloud. The ego estimation of the camera can be improved by using frame-point cloud registration method. The efficiency and accuracy of the pose estimation are improved, and a drift-free camera tracking is achieved.In order to identify and understand the real objects in the assembly environment, a registration based 3D perception method is proposed. By transforming a virtual model’s point cloud to align real scene’ point cloud, identification and perception of a real object in the assembly environment is defined using similarity evaluation. Aim at reducing the influence of different density of point clouds on the aligning accuracy, an adaptive density registration method is introduced. By employing weighted-voxel sampling step to obtain point clouds of the scene and virtual model uniformly, real objects in scene can be identified using the affine invariability of a four congruent points set. The method can reduce large amount of the point cloud and the time needed to test object candidates can be saved greatly, meanwhile the registration performance is improved. The experiments show that the improved algorithm can accurately identify the common machinery objects in clutter assembly environment and can estimate their pose, which is suitable for augmented reality interaction application under noisy condition.A monocular augmented reality based experimental platform is developed. The platform is oriented to assembly tasks, providing texts,2D images and 3D models to the users during the assembling process, helpings users understand the assembly task and gain assembly operation experience. Assembly sequences and process planning documents can be created based on the assembly content. The platform employs scene reconstruction, camera tracking and 3D perception technology. The platform verifies the real-time performance, robustness and accuracy of the presented algorithm. The issues of pose estimation of machinery parts and 3D immersion can be addressed effectively.