Research On Occlusion Handling Between Virtual And Real Worlds In Augmented Reality Systems

People pay more and more attention to augmented reality (AR) along with the development of information technology. Its range of application expanded significantly recently. AR is a technology that superimposes the computer generated virtual objects, scene and information to the real world. It expands and augments the real world with additional information. The AR has close relationship to virtual reality (VR), but they have distinctions. All the objects in the VR system are computer generated. Different with the VR system, AR system emphasizes the fusion between the virtual objects and the real worlds.The early researches on augmented reality mainly focus on the registration between virtual and real worlds and the 3D reconstruction technology. People have made significant improvements in these areas. With the development of research, occlusion handling has become another crucial problem we need to solve. When we overlay the virtual objects on the real scene, there are spatial relationships between the virtual and real objects. Sometimes the virtual objects occlude the real object and sometimes on the contrary. At present, most AR systems simply draw virtual objects on the composite images regardless of the spatial relationships between the virtual and real objects. This leads to that the real objects are occluded by the virtual objects all the time. It will make the user difficult to get the correct spatial relationships between the real and virtual objects. Focusing on this problem, we analyze the advantages and disadvantages of the existing approaches. Then we take full advantages of digital image processing, pattern recognition, computer vision and non-linear optimization technology to investigate the real-time semi-automatic occlusion handling method, real-time automatic occlusion handling method and non-rigid occlusion handling method. The goal is to get correct spatial relationships between real and virtual objects in real time and improve the availability of AR systems.The main studies and achievements of the thesis are listed as following:(1) A new framework handling occlusion problem based on contour trackingIn the existing model-based occlusion handling approaches, the 3D models of the real scene or the occluding real objects need to be reconstructed. This is difficult in the complex scene. And the existing depth-based methods are only suitable to the static scene. To enlarge the application range, we propose a novel framework based on contour tracking to handle occlusion problem. Our approach can successfully handle the occlusion problem without 3D reconstruction and is suitable to the case when the camera changes over a wide range of viewing angles and volumes.(2) A real-time semi-automatic occlusion handling methodWe propose a novel real-time semi-automatic occlusion handling method. Our method is divided into three steps:occluding real object specifying, object tracking and occlusion handling. In occluding real object specifying step, user specifies the occluding object using interactive segmentation method. Then the contour of the specified object will be tracked in the subsequent frames using the fast level set contour tracking method which is accurate and real-time. In occlusion handling step, all the pixels on the tracked object are redrawn on the unhandled augmented image to produce a new synthesized image in which the relative position between real and virtual objects is correct. The proposed method has several advantages. Firstly, it is robust and stable as it remains effective when the camera is moved through large changes of viewing angles and volumes or when the object and the background have many similar colors. Secondly, it is fast as the real object can be tracked in real-time. Lastly, the seamless mergence is improved by utilizing smoothing technique. Several experiments are provided to validate the performance of the proposed method.(3) A real-time automatic occlusion handling methodTo satisfy the requirement of minimizing the human intervention in AR systems, we propose a new real-time automatic approach that resolves occlusion problem. The main interest is that it can automatically obtain the proper spatial relationships between virtual and real objects in real time. Our approach is divided into two steps:off-line disparity map constructing and on-line occlusion handling. In the off-line stage, the disparity map of the real scene is constructed using the global stereo matching method prior and then the disparities are refined by means of the fast mean shift method. Because the depth values of objects in different positions are different, the real object that occludes the virtual object can be specified according to the depth value. In the on-line stage, the contour of the specified object is tracked using the real time object tracking method with the combination of feature tracking method and minimum s-t cut method. The augmented image with correct occlusions is produced by redrawing all the tracked object pixels on the augmented image. Compared with the existing methods, the proposed approach can automatically resolve occlusion problem in real time. The effectiveness of our method is demonstrated with several experimental results.(4) An occlusion handling method for non-rigid objects in augmented reality systemsThere are always non-rigid objects existing in the real world. Different with rigid objects, the shape of the non-rigid objects will change randomly. So the general occlusion handling methods used for rigid objects are not suitable to the non-rigid objects. We propose an occlusion handling method for non-rigid objects in augmented reality systems based on active appearance models (AAM) and factorization method. This method can not only calculate the extrinsic parameter but also solve the non-rigid occlusion problem.