Researches On The Method Of High Quality Virtual View Synthesis

Multi-view video produces multiple viewpoints on the client which can provide a rich visual experience, but it increases the difficulty of video information capture, compression, and transmission. It is unrealistic to transmit unlimited number of viewpoints to meet client’s demand, thus it requires virtual viewpoint rendering technology to render multiple viewpoints on the client side using small amounts of viewpoints. The quality of synthesized view determines the quality of multi-view video system, so virtual viewpoint rendering technology is one of the critical technologies in the Multi-View video system. The depth image based rendering is currently widely used, which includes 3D-warping, hole-filling, image fusion and other processes. This paper focuses on the three processes to analyze the deficiencies of view synthesis reference software, and proposes improved algorithms.(1) The fusion process of VSRS only uses a single weight, so the virtual viewpoint can not achieve the best quality. An image fusion method based on machine learning is proposed. The eigenvalues and the labels obtained from the process of view synthesis are trained to get the classifier model which is used to calculate the weights of the pixels. Experimental results show that the objective quality can be improved up to 0.17 d B and it is effective in the case of large distance between the virtual and real cameras. The subjective quality has also improved.(2) Due to inaccurate depth information, 3D-warping will generate error projection and the quality of virtual viewpoint will be low. An advanced method of 3D-warping is proposed. Two points are mapped to virtual viewpoint at a time, and then the integer pixels selected between two mapped points are interpolated by proposed interpolation algorithm. Experimental results show that the objective quality can be improved up to 0.94 d B, and the subjective quality can be greatly improved.(3) If the depth information is not accurate, the holes will be shifted in the virtual viewpoint rendering process. The pixels around the edge of hole will be mapped inaccurately. A hole-filling algorithm taking the information around hole into account is proposed. The holes are widened according to the color and depth information around and eventually these widen areas are filled according to another reference viewpoint. Experimental results show that the improvement of objective quality is not obvious, but for some sequence the subjective quality has improved. The filter method of the VSRS for filling small holes is simple, and in some regions the subjective quality of virtual viewpoint has dropped significantly. Therefore an improved method for small holes filter is proposed. The information around the holes is used to fill holes, and the holes will be smaller and smaller. Finally the holes will be filled completely. Experimental results show that the proposed method can be used to fill small holes, and the objective quality of virtual viewpoint is improved, and the burr synthesized by VSRS is also removed.