Research On Technology Of Depth Video Holes Repairing

Depth video technology has been applied to free viewpoint video,3D scene reconstruction,robot visual map,visual reality and other popular areas.It reflects the actual distance of scene depth information which gives the users a great visual shock.The methods of depth video acquisition include software estimation and depth camera capture.Software estimation method not only has high calculation complexity,but also the estimated depth information is not accurate.With the development of the depth camera technology,it has become a hot research topic of the depth camera acquisition technology due to the advantage of low cost.The depth camera obtains the depth information mainly depending on the infrared laser technology.First,the infrared light is transmitted to the scene through one sensor,and the speckle pattern is received by another sensor.Then,the scene distance is calculated by the speckle pattern and mapped to the pixel range of the image which form the final depth map.However,due to the influence of reflection of scene surface,refractive index and occlusion exposed,the depth information captured by depth camera will lose information and generate holes on depth map.The loss of depth information will seriously affect the quality of 3D multi-view rendering and scene reconstruction,which causes uncomfortable visual experience for users.This thesis carries out research on depth video holes' repair technology to improve the quality of distortion depth video,which mainly includes the following three parts.(1)Traditional image restoration methods by using spatial correlation restoration could not effectively repair depth distortion boundary.In this paper,it is founded that there is texture and structural similarity of depth image region corresponding to similar color image due to the relationship between the color image and the depth image of the same scene.Therefore,a repair strategy based on super pixel segmentation method is proposed to improve the accuracy and precision of the repair of the holes.The regions corresponding to missing pixels ratio is divided into four categories,respectively,not any empty area,small holes area,big holes area and whole holes area.The four regions are used for different suitable holes repair strategies.The experimental results show that,the average RMSE of the proposed method are reduced by 2.9584,0.8229 and 0.0780,compared with Telea's,Shen's and Scheming's methods.Besides,the subjective Quality is also improved.(2)Many depth repair methods mainly used effective depth information,but the scene sparsity of depth information was not fully considered.It is considered that the distortion of depth information captured by the current depth camera is related to none distortion of depth information so the sparse distortion model is established in this article.According to this model,the undistorted depth matrix and distorted depth matrix was represented by sparse distortion respectively.The effect of noise sparse matrix will be reduced by a joint structure filtering method as much as possible,and the original pixel's value was estimated by the depth extraction factor judgment matrix to repair depth maps.The experimental results show that the subjective quality and objective quality are better than other methods.In addition,the subjective quality of virtual viewpoint rendering is better than existing methods.(3)Considering the correlation between color and depth video alignment frames,we proposed a static spatial and temporal consistent depth video inpainting algorithm.Firstly,the static and motion regions of depth video frame are separated by extracting the moving target,and the background subtraction extraction method is improved in this article.In order to eliminate the influence of similarity of color texture region between foreground regions and static regions,the depth difference in background samples is calculated in updating process.Secondly,the depth holes on static and motion regions is filled by static region cross repair method and static joint weight filtering method respectively.Finally,two regions repairing results are merged and color information to guide the depth edge filtering is used.The experimental results show that the proposed method achieves better performance than the existing depth video repairing methods.