Depth Image Enhancement Algorithm And Its Application In Free Viewpoint Video

The vigorous developments of computer vision technology, video technologyand network technology provide the possibility for the development of free viewpointvideo (FVV). The one of the key points of FVV is how to represent3D geometry ofreal scene (3D representation). At present, there exist two kinds of3D representations:a) Multi-view based3D representation; b) depth image based3D representation. Theformer can render high-quality image of virtual viewpoint by capturing3Dinformation with camera array. However, the way of high-density sampling consumesa lot resource of codecs and network bandwidth. The second solution uses colortexture plus depth image representing3D information of real scene, which greatlyreduces the bandwidth requirements and there is no need of high-density samplingdue to depth image can describe the3D geometry. Additionally, the method cangenerate realistic image of arbitrary virtual viewpoint in a limited range. Therefore,Depth image based3D representation is generally considered as the alternativescheme about the next generation of FVV due to advantages of low networkbandwidth, high speed of virtual perspective drawing and realistic rendering.However, obtaining high-quality depth is quite challenging. To address this problem,our papers focus on depth image enhancement algorithm research and achieved thefollowing results:Firstly, we proposed a depth image enhancement algorithm based on improvingbilateral filtering. The current depth of image has two main methods: passive opticalmethods and active sensor. The passive method is to obtain depth image throughcalculating the features corresponding relationship between2D image and3dgeometry by computer vision technology. Passive optical method requires high timecomplexity, strict constraints and accurate image correction etc. Therefore it isdifficult to be applied in practice. Active optical method captures the depth informfrom3D real scene using time-of-flight (TOF) or light coding. Compared to passivemethod, active method becomes an increasingly concerned method due to its real-time,accuracy and practicality. However, the low-cost sensor has the drawback of lowresolution, cavity, unmatched boundary of edge, limiting its application in practice.Therefore, the paper uses joint bilateral filtering interpolation to enhance depth imagethrough fusing high-resolution color image. The results of this paper method improved increased the MSE by3.23%in average which has better effect inedge-preserving. For the existing joint bilateral interpolation method easily makesdepth image edges much smooth, this paper improved the existing joint bilateral depthinterpolation method by taking guided depth similar item and gradient item intoaccount. Experiments prove that compared to the existing joint bilateral interpolation.Secondly, for the low-resolution, cavity, unmatched boundary of edge of depthimage captured from Kinect, we proposed a depth image enhancement method thatfuse color image by combining depth-based foreground segmentation and jointbilateral interpolation, and applied the enhanced depth image to FVV. Our method notonly can effectively fill the hole and solve the unmatched foreground edge, but alsoenhance the resolution of raw depth image. Moreover, the enhanced depth map iseffectively applied in FVV.