Robust Stereo Matching With Surface Normal Prediction

Depth recovery is a fundamental problem in the field of computer vision.Binocular stereo matching is one of its most useful methods,and with a great theoretical research value and a wide applied prospect.Traditional stereo matching approaches generally have problems in handling textureless regions,strong occlusions and reflective regions that do not satisfy a Lambertian surface assumption.These regions lack sufficient brightness or color information to be matched correctly,or to get depth recovered accurately.In this paper,to overcome these inherent difficulties in stereo matching,we propose a stereo method to combine with predicted surface normal by deep learning.We can faithfully convert the predicted surface normal map to a disparity map by solving a least square system with both normal and depth constraints.To achieve this goal,we design a disparity confidence measurement based on plane fitting assumption.Constraints on the absolute value of the depth are established by selecting reliable disparities with this measurement.Besides,we also bring up an edge detection and fusion algorithm to locate depth discontinuity.These edges help manage normal constraints and maintain discontinuity on edges and continuity on other regions.Then we refine the disparity map iteratively by bilateral filtering-based completion and edge feature refinement.We also propose a fast version of our method,which could compute one frame or more per second.Experimental results on the public dataset and our own captured stereo sequences demonstrate the effectiveness of the proposed approach,which can significantly improve the disparity estimation result especially in the occluded and textureless regions.Our algorithm has great expansibility and universality.It could work with multiple other stereo matching algorithm and refine the depth results.