Depth Estimation From Catadioptric Omni-directional Images

Depth estimation is one of the key problems in the field of computer vision. With the rapid development of omni-directional imaging technique, depthe esitimation from omni-directional image, which can be widely used in applications such as large FOV 3D reconstruction, robot navigation and object detection and tracking, is becoming a new hot topic.Omni-directional system has a different imaging model from traditional ones. Objects are serious deformed in omni-directional images. Consequently, existing depth estimation methods, which are mostly devised for perspective images, are not appropriate to apply to omni-directional images. In addition, omni-directional systems have different structures due to different configurations of reflective mirrors and CCD sensors, and a specific depth estimation method should be devised for a certain system structure. Therefore, in this paper, by taking the characteristics of omni-directional image into consideration, we propose three different depth estimation methods corresponding to three different omni-directional systems. Our work mainly includes:1. Detailed analysis about depth estimation from omni-directional image. Beginning with defining related concepts of omni-directional system, we give an exhaustive survey and taxonomy about depth estimation from omni-directional images. Methods of omni-directional image unwrapping, omni-directional system calibration and depth estimation are investigated and analyzed. Especially, we discusse approaches that recover depth information from single omni-directional image, which are not studied much in literatures.2. Depth estimation from stereo SVP omni-directional images. Single view-point (SVP) catadioptric omni-directional image is attractive since it can be unwrapped into cylindrical panorama or perspective image, and can be then dealt with traditional imaging processing methods. Therefore, it is reasonable to estimate depth information of scenes from two stereo SVP omni-directional images captured at differet positons. In this paper, we focus on two key problems of this mthod: omni-directional image rectification and stereo matching. We first design a SVP omni-directional system and use it to obtain stereo images via moving it horizontally. Then, a rectification approach for catadioptric omnidirectional image pair is proposed. To reduce resampling-effect and image deformation resulted from image rectification, problem of how to sampling reference images maximum as much as possible according to the epipolar geometry is studied and solved. As to stereo matching, a novel fast algorithm via semi-global energy optimization is developed. It constructs an energy function based on color consistency and restrictions between region boundaries, and the pixels involved are only those positioned on region boundaries, resulting in greatly reduced vertex number in the constructed graph and subsequently improved efficiency.3. Depth estimation from stereo non-SVP omni-directional images. Lliterature proves that an efficient SVP of omni-directional system can only be achieved with precisely aligned mirrors of parabolic or hyperbolic profile. This enforces rigorous restriction on the configuration of camera and mirrors. In fact, some other profiles, though do not have the SVP property, are desirable for certain reasons such as cheaper cost and more practical implementation. Therefore, in this paper, we propose both a typical non-single view-point (non-SVP) omni-directional stereo sensor and its corresponding depth estimation method based on graph-cuts optimization. The sensor comprises a perspective camera and two separate reflective mirrors that could be any radially-symmetric ones. To formulate the depth estimation more consistent with proposed sensor, we divide the depth space of scenes with a sequence of virtual coaxial cylindrical layers, and model depth estimation as a labeling problem. In the labeling procedure, by considering the characteristic of omni-directional image, we further devise novel neighborhood systems and smoothness constraint which perform better than traditional ones.4. Depth estimation from a single non-SVP omni-directional image. Different from traditional 3D reconstruction via stereo vision, this paper studies the reconstruction of straight horizontal lines in 3D space from single 2D omni-directional images. It demonstrates that, for symmetric non-central catadioptric systems, the equation of a 3D horizontal line can be estimated using only two points extracted from a single image of the line. By exploiting the peculiar property of horizontal line image in catadioptric system, line detection from single omni-directional image can be simplified. Meanwhile, a horizontal line reconstruction algorithm based on main-point image and non-main-point image is developed. We also evaluate how the precision of feature point extraction can affect line reconstruction accuracy. Detailed experiments justify that, the proposed method is suitable for 3D reconstruction of man-made scenes with abundant straight line features such as outdoor street blocks and indoor structural enviroments.