Research On Virtual View Rendering For FVV

With the advent of computer graphics, computer vision, multimedia technology and related fields oftechnology, free viewpoint video (FVV), the next generation of new media technology, has developed rapidly.FVV allows for interactive selection of view position and orientation within a certain range. Since it can providedepth perception of the observed scene, FVV has attracted significant interests from both industries and academiawith a variety of applications. Video data acquisition, data transmission and virtual viewpoint video rendering arethe three important parts of FVV system, which are closely related to depth maps. Currently, depth video can beobtained via depth camera and depth estimation software, both of which cannot yield sufficient accurate andconsistent depth video. The inaccuracy and inconsistency of depth video lead to pixels projecting to the wrongplace, which may introduce geometry distortions in the synthesized views. Typically, abrupt depth variation easilycauses the foreground object projected to background regions. As a result, annoying effects such as artifacts,“rubble”effect and even holes, severely decrease the quality of synthesized views. Therefore, depth videos areimperatively needed to be pre-processed before transmission. Because of a vast amount of data to be transmittedto the users in FVV system, efficient compression techniques are essential. However, high compression ratioinevitably brings about distortions in depth maps. The impact of depth coding distortion on the virtual view pointis a challenging problem to be solved. To address key challenges such as pre-processing, compression andreconstruction of depth video in FVV system, effective researches on virtual view rendering technology have beenconducted in this paper for the propose to raise the quality of the synthesized views.To reduce the geometry distortions caused by spatial inconsistency of depth maps, an abrupt variationdiminish method is proposed to refine depth maps in this paper. The relationship between hole’s size and depthvalues difference between two horizontal adjacent pixels is devised as a model. Abrupt variation regions aremarked. The depth value difference is gradually made smaller with adaptive step width. Experimental resultsshow that, compared with the method based on original depth videos, the objective quality of virtual viewsgenerated by proposed method is improved by0.03~0.14dB while the blur on object edge is repaired and the edgebecomes distinct.A depth video transmission method is proposed to solve a series of problems such as huge data amount ofdepth video and effect of depth video coding distortion on virtual view rendering. This method solve the aboveproblems from three aspects, that is, depth video pre-processing, depth coding and depth video pro-processing.The common methods in depth video pre-processing are smoothing the whole depth maps with a certain filter orsmoothing partial regions while protecting edges and so on. All of these methods do not consider the influence ofdepth value change on rendering. The proposed method first combines3D-warping procedure to obtain the depthvalue variation range which would not lead to distortion. Then the range is utilized to aid Gaussian filter.Compared with Zhao’s method, the proposed method is robust and suitable for both series and parallel sequenceswhile Zhao’s method is only fit for parallel sequences. In order to reduce the bandwidth for transmission and thecomplexity for coding, the above pre-processed depth videos are down-sampled. However, depth videos loss lotsof information due to down-sampling and coding quantization. Therefore, the proposed transmission methodreconstructs the decoded and up-sampled depth videos adaptively. An adaptive depth edge reconstruction model is applied in the last step of the proposed method. It is designed based on three factors: frequency, similarity anddistance. In the model, the proportion of three factors is adjusted adaptively. Experimental results show that thewhole proposed transmission method can save bit rate ranging from53.87%to73.62%. As it can properlyreconstruct the sampled depth videos, the proposed method also improves the overall rendering quality.