Research On Depth Enhancement Based On Image Inpainting And Filtering

The proliferation of computer vision and robotic applications has made depth-based applications more and more extensive,and the quality requirement for depth information has become higher and higher.However,the images acquired by various types of depth sensing principles,such as binocular stereo vision,structured light,TOF,etc.are often incomplete,and there are often problems such as lackness of depth value,noise,and low resolution.In this paper,two enhancement algorithms are proposed to solve the above-mentioned problems in the captured depth images: depth enhancement based on inpainting and filtering,and depth enhancement based on object continuity.In the study of depth enhancement algorithm based on inpainting and filtering,the propagation sequence of traditional inpainting algorithms,which is applied to depth enhancement,is improved.Afterwards,two filtering methods combining three-side information,are proposed to remove the noise and artifacts in the inpainted images.In this paper,the decision process of the inpainting sequence is divided into two steps: determining the candidate set and selecting the target pixel.The candidate set is used to determine the range of pixel points to consider.The order of the target pixels in the candidate set is determined by the priority evaluation function which is defined by confidence and orientation factor.In addition,the incompleteness of depth image and the inconsistency between depth and color image result in the unreliability of the use of unilateral pixel similarity.In the improved method,the depth pixel and color pixel similarity is combined together with the distance factor,which ensures the local relevance in the image.In the research of depth enhancement algorithm based on object continuity,according to the continuity of the object inside the depth image,the image is divided into objects.For each object,the 3D regression model is built using its internal known information.The resulting model is used to predict the depth value of missing pixels.According to the integrity of object existed in the real world,the change of the z value in the camera coordinates continuous,so the corresponding depth values in image coordinates also changes continuously.In this paper,we call it object continuity.Based on this knowledge,this method uses the quadric surface as the model to be constructed,and solve the parameters by minimizing the loss function.The holes in each object are iteratively filled with this algorithm.Finally,the depth enhancement algorithms presented in this paper is evaluated on the public datasets subjectively and objectively.First of all,the two enhancement algorithms proposed in this paper are tested and evaluated separately,including the comparison and evaluation between multiple alternatives in single-step process and the performance of the algorithm itself on the whole dataset.After that,visual and quantitative comparisons of the two algorithms proposed in this paper and three other state-of-the-art depth enhancement algorithms are made on the test set.Visual comparison shows that the algorithms proposed in this paper can obtain better visual effects,especially for structural edges,more structural information is preserved and less blurring is produced.In terms of quantitative comparison,the proposed algorithms improve both the SSIM and PSNR value comparing with the three other methods.The average PSNR has increased by 0.8-2.0d B and the average SSIM increases 0.03-0.24.In addition,the SSIM values obtained by the two proposed methods mostly distributed above 0.9.