Research On Virtual Viewpoint Synthesis For Free Viewpoint Video

As people’s demands for interactive and immersive multimedia increase,free viewpoint video has received a lot of attention.Free viewpoint video is an interactive video(or 3D video)playback mode,where users can select the video playback position and angle.To achieve this effect,a large amount of viewpoint information needs to be captured from the video scene,which can lead to high costs.Therefore,free viewpoint video uses Depth Image Based Rendering(DIBR)to reduce the collection of viewpoint information.However,when synthesizing new viewpoints from known viewpoints through DIBR,problems such as cracks,artifacts,and holes may occur,which can result in poor image quality of virtual viewpoint images and a decline in user experience quality.Therefore,this paper focuses on the research of many problems caused by the exposure of holes in the occlusion area and image edge loss caused by viewpoint differences in depth image-based rendering,aiming to improve the quality of virtual viewpoint images.The main work is as follows:(1)A virtual viewpoint synthesis algorithm based on background reconstruction is proposed to address the issue of hole filling.Firstly,the color image and depth map are segmented by different methods,and the foreground binary map is obtained by combining the two,followed by the construction of a stable background sequence using Gaussian Mixture Model(GMM).Secondly,the depth prediction is performed on the background depth map with holes to ensure that the average depth of the best matching block in the color image is not significantly different from that of the target matching block.Then,the Criminisi algorithm is improved by incorporating depth information and the color information of the previous frame to fill holes in the color background.Finally,the color background is used to fill holes in the virtual color image.Through experimental comparisons,the algorithm reduces foreground bleeding in virtual synthesis,ensures high temporal consistency of the same hole repair area in adjacent frames,and improves the subjective perception quality of virtual viewpoints.The algorithm shows an average improvement of 1.58 d B,1.74 d B,0.97 d B,and 0.31 d B in Peak Signal-to-Noise Ratio(PSNR)compared to VSRS method,Criminisi method,Ahn method,and Luo’s method,respectively.There is also a significant improvement in Structural Similarity Index(SSIM).Meanwhile,the results of Frame Differential Flicker(FDF)comparison also show that the algorithm has better temporal consistency in hole repair areas.(2)A virtual viewpoint synthesis algorithm based on generative adversarial networks is proposed to address the issue of hole filling.Firstly,this method uses an edge prediction network to predict the edges of the holes in the target view,and uses these edge information as guidance to help the hole filling network learn more targeted filling.Secondly,the virtual color image,predicted edge image and corresponding mask image are used as input to the hole filling network.The encoder extracts the structural and texture features for multi-scale hole filling.The texture and structure feature fusion module is then used to integrate the texture and structure features into a whole.Then,a context,channel and spatial feature aggregation module is used to enhance the correlation between local image features and allocate channel weights to ensure the consistency of the hole boundary and internal features.Finally,the context,channel and spatial feature aggregation output is used to supplement the decoder information,and the virtual color image after hole filling is obtained.Through experimental comparisons,this method improves the PSNR by an average of 1.97 d B,3.27 d B,1.09 d B,0.22 d B,1.45 d B,and 2.18 d B compared to the Ahn method,Criminisi method,Luo method,MEN method,LGN method,and CSA method,respectively.Additionally,there is a small improvement in SSIM.The FID results indicate that the virtual color image generated by this method is more similar to the real image,and the subjective visual quality of the virtual viewpoint image is improved.