| Omnidirectional depth estimation systems are superior to conventional stereo systems because they allow us to identify objects of interest in all directions without any blind spots.With the improvement of image quality and field of view demand,wide baseline fisheye cameras are widely used in many fields,but the images captured by fisheye cameras have severe distortion,which makes it impossible to apply stereo matching algorithms directly to fisheye images as ordinary fluoroscopic images.In this paper,we explore the imaging laws of fisheye cameras and design a stereo matching algorithm based on binocular fisheye cameras.Then,we use deep learning tools to learn the end-to-end mapping from fisheye images to disparity maps,and learn and predict the omnidirectional disparity output of the fisheye camera directly from the input image data.The main research components are as follows.(1)We perform calibration work on the fisheye camera,obtain the internal and external camera parameters for the projection process of the fisheye camera,and use the reprojection error to measure the accuracy of the calibration results.Based on the calibration data,the pair of polar constraint curves of the fisheye images are derived,and the polar curves are used as search paths to limit the disparity range,which provides favorable conditions for the stereo matching algorithm to be directly applied to the fisheye images.(2)To address the problem that the cost aggregation scheme with local square windows in the traditional stereo matching algorithm cannot apply the fisheye sphere imaging model to obtain a correct match,we propose a minimum spanning tree based stereo matching algorithm for fisheye images.The algorithm transforms the fisheye image matching problem into a tree node matching problem,and the non-local matching scheme of minimum spanning tree(MST)discards the cost aggregation idea of local square window and solves the problem of large stereo matching error caused by the distortion of fisheye image.By applying this stereo matching algorithm to the fisheye image projection model,the fisheye image matching results with global nature can be obtained.(3)We propose an end-to-end convolutional network to predict disparity information in fisheye images.For the Focus 3D real dataset,the network uses a "matching layer" to match the correlation between the left and right feature maps.For the OmniThings synthetic dataset,the network adds a multi-scale learning module to learn more fully the feature information of the images in the multi-scale space.The fisheye image features are obtained using an encoding-decoding structure to generate omnidirectional disparity estimates.By validating the network structure on the OmniThings synthetic dataset and the Focus 3D real dataset,the experiments show that the end-to-end convolutional neural network proposed in this paper can effectively predict the omni-directional disparity of fisheye images,and the error rate is better than other network algorithms. |
