Research On Depth Estimation And Refinement Under Micro-baseline Inputs

As the pre-commerial services of 5G network begins,high transmission speed of the network provides terminal devices with comutation and storage resources of the cloud,which supports the development of vision applications in terminal devices.3D information acquirement is the basis of many vision applications,including those ones in smartphones.Smart phone manufacturers enable motionless phones of 3D perception by dual cameras or depth camera.Apple and Google have released Augmented Reality SDK to get structure information by moving phone cameras.However,narrow baseline of dual cameras and small motion of phones raise the uncertainty of depth estimation,hence makes it more difficult for phones to estimate scenec structure information.The research topic of this thesis is depth estimation under micro-baseline inputs,which overcomes the effect of micro baseline inputs and produces better depth results in accuracy,robustness and computation speed.In computer vision,baseline refers to the distance between optical centers of camera.The uncertainty of depth estimation increases with the descrease of baseline by square growth.As for stereo problem,micro-baseline inputs will raise the required precision.As for multi-view problem,micro-baseline inputs will disable epipolar geometry algorithms.Therefore,structure information estimation under micro-baseline inputs is a particular case.Commonly used methods and frameworks in Stereo Matching,Sturecture from Motion and visual Simultaneous Localization and Mapping can not be applied directly under micro-baseline inputs.It needs sepcial consideration and independent design apart from general structure information estimation problem.In consideration of the specularity of micro-baseline inputs,depth estimation methods tailered for stereo and multi-view images with micro baseline are proposed in this thesis,as well as the actual results in some related applications.The proposed methods are combination of currently popular learning based methods and traditional vision methods,which not only overcome inherent shortcomings of traditional methods,but also surpass state-of-art methods under micro-baseline inputs.The main contrubitions and innovations are summarized as below:1.A phase-based regression methosd are proposed for disparity estimation problem under micro-baseline inputs.Different from commonly-used means of cost computation and aggregation in stereo matching problem,proposed method transforms stereo matching problem into phase regression problem in frequency domain.After decomposition of complex steerable pyramid,a set of phase maps with several orientations and scales are selected.And then learning methods are used to utilize redundancy of phase information and local gradient information for regression learning.The combination of domain transform and machine learning elevates the precision of traditional phase-based methods to the next order of magnitude,and transcends state-of-art stereo matching methods under micro-baseline stereo input.2.For micro-baseline muti-view inputs,two different methods are proposed to deal with the failure of epipolar geometry methods caused by uncertainty of depth estimation.One proposed method of depth estimation under micro-baseline multi-view inputs is joint optimation based on point-and-line features.Point features are difficult to detect in textureless region,so line features are added into the framework to enhance the robustmess of system and improve the prediction at edge.Besides,a propogation based sparse to dense method is proposed,which cuts plane sweeping time by half with the same quantization accuracy.In general,the proposed method has the best performace among all compared methods.3.Another proposed method is to estimate relative depth map by parallel reference plane.A few modification has been made to adjust original plane parallax framework to micro-baseline inputs.Firstly,simplify presentation and solving process of camera motion.Secondly,minimize coordinate transformation error and patch-based photometric error to determine points and motion of reference plane.Besides,a geometry similarity based meaturement is proposed to ensure the parallel of reference depth plane and imaging plane.And ratio of eigenvalues is used to validate optical flow result.The operations together guarantee accuracy of optical flow and depth-plane-based motion parallax,hence quality of relative depth map in reference view.The proposed method is suitable for scene with planes,and biggist advantage of this method is short calculation time.4.A hybrid depth refinement framework is proposed for depth impaiting,super-resolution and edge refinement.It not only depends on color information,but also extracts various information from low-resolution depth map as different inputs of conditional random field:nearest neighbor interpolation is used to define prior states potential,which keeps total label number of conditional random field.Color information as well as depth produced by convolutional neural network are used to define transform function.Bilinear interpolation is used to get depth gradient,which produces mask defines updateing regions.The proposed method conbines CNN’s ability of non-linear mapping and CRF’s character of discrete optimization to achieve depth super-resolution.Experiment result shows that the proposed method has clear advantage in exactitude of edge.