Research On The Quality Enhancement Of Virtual View Synthesis

With the rapid development of information and computer technology,people are not satisfied with the visual experience of two-dimensional video.As a new generation of dig-ital video technology,3D video and free viewpoint television(3D Video/Free-viewpoint TV,3DV/FTV)can provide a very wide viewing degree of freedom,the immersive vi-sual effect,and flexible interaction function.For these reasons,it has attracted extensive attention both in academia and industry fields.However,the advantages of the degree of freedom and immersion of 3D video are at the cost of huge video data,which poses a serious challenge to the capture,storage and transmission of 3D video.Recently,the format of Multiview Video plus Depth(MVD)has been established as one of the main representation methods for current 3D video by MPEG internation-al organization for standardization.The technology of Depth-Image-Based Rendering(DIBR)can effectively reduce the amount of 3D video data,and can meet the needs of user's stereoscopic visual experience.However,limited by current performance of depth camera and transmission bandwidth,there generally exists the mismatch problem of res-olution between the depth map and its corresponding texture image.In addition,due to the influence of noise during the capture,encoding and transmission on depth map,it will inevitably deteriorate the quality of the virtual view and the effect of the user's visual experience.Therefore,it is of great theoretical significance and practical value to study on the quality enhancement of virtual view image.This thesis focuses on two main issues about 3D virtual view quality improvement,i.e.,resolution enhancement for depth map and hole filling for virtual view synthesis.The main contributions are as follows:· A resolution enhancement algorithm based on fractal transform for depth map is proposed.Limited by channel transmission bandwidth and acquisition performance of depth camera,it is difficult to directly get high-quality virtual view image vi-a high-resolution texture image and low-resolution depth map.Based on fractal theory,the fractal parameters are extracted to describe the self-similarity of depth map.Then they are favorably formed the resolution independent representations.Meanwhile,the reconstruction of the depth maps with multiple resolutions is im-plemented by the fixed point iterative system.Hence,high-quality depth image can be obtained by further fusing the reconstruction residual of the depth map with low resolution.Compared with traditional interpolation algorithms,the experimental results have demonstrated that the proposed algorithm obtain 0.6 dB gain for vir-tual view with zoom factor 2 and 6%increase for virtual view image with zoom factor 8.· A resolution enhancement algorithm with structure preserving for depth map is proposed.Based on the local fractal analysis on gradient field,it is useful to avoid edge degradation in the process of resolution enhancement for depth map effec-tively.Considering the contour correspondence between depth map and texture image in multi-view video,the edge consistency constraint is enforced to ensure the structure preserving up-sampled of low resolution depth map.In addition,the complexity analysis of the proposed algorithm is also given theoretically.Exper-imental results have shown that the gain of 0.25-1.46 dB and 0.21-0.80 dB can be obtained,respectively,in view of the qualities of depth map and virtual view.Meanwhile,the corresponding subjective quality has also been evaluated for the effectiveness of the proposed algorithm.· A hole filling method based on Generative Adversarial Networks(GAN)is pro-posed.Due to the discontinuity of depth values,the holes always appear around the edge of objects in the virtual view image.Based on edge sampling on the in-put image,deep convolution generative adversarial networks(DCGAN)is applied to learn the mapping of low dimensional hidden space to real sample space.To ensure the authenticity of the generated image,the consistency of non-hole region and the continuity of local texture at the same time,an unified constraint is utilized by combining several loss functions that include the adversarial loss,contextual loss and depth loss.Thus,the optimal factor,which represents the corresponding hole region of the virtual view image in the hidden space,is obtained and then vir-tual view image with high quality will be synthesized.The experimental results have demonstrated that the proposed method can effectively fill the hole region and achieve better virtual visual effect than the comparison algorithms.