Research On Deep Learning Based Virtual View Synthesis

Virtual view synthesis refers to the process of synthesizing a new view images of a scene using one or multiple views of images.This technology is widely used in 3D video,virtual reality,robotics,and autonomous driving.Among them,Depth Image Based Rendering(DIBR)is an essential virtual view synthesis technique.DIBR uses the color image and corresponding depth image of the reference viewpoint to generate virtual view images.It has advantages such as fast rendering speed,strong realism,and low requirement for the number of reference viewpoints.However,in the 3D warping process,the pixels of the reference view images are warped to the fractional positions in the virtual view images,resulting in geometric warping errors.Geometric warping errors result in pixel or foreground object displacement,thereby seriously reducing image quality.This thesis focuses on the problem of geometric warping errors.To address the issue of geometric warping errors,this thesis proposes a geometric warping error aware virtual view synthesis algorithm.The algorithm consists of three modules,which are used to improve 3D warping,view blending and hole filling.Firstly,the geometric warping error aware alignment module can align the displacement pixels to the correct position.Based on the deformable convolution,the module uses the geometric warping error information to learn the offset of the deformable convolution,so as to achieve alignment and enhance the warping process.Secondly,the view blending module based on attention mechanism can improve the quality of view blending.In the process of view blending,the quality of the warped pixels from different reference viewpoints is different.This module can output the quality score of the pixels,thus adaptively fusing the pixels from different reference viewpoints and enhancing the view blending process.Finally,a partial convolution based hole filling and refinement module is proposed to fill the remaining holes after view blending and improve the overall image quality.In addition,temporal enhancement can reveal the real texture of hole areas through multiple frames,enhance image details,and improve image quality.Therefore,this thesis further proposes a temporal enhanced virtual view synthesis algorithm.After pixel alignment,view blending,and hole filling,the proposed deformable convolution based temporal multiframe enhancement module is used to perform time-domain enhancement of single-frame images.It uses the image information of adjacent frames to learn the offset of deformable convolution and captures the most relevant image texture features of the auxiliary frames(previous and subsequent frames),so as to enhance the details of the target frame and further improve the image quality.In summary,this thesis mainly studies the problem of geometric warping errors in 3D warping,which is used to enhance the process of 3D warping,view blending and hole filling.On this basis,this thesis proposes an effective temporal enhancement scheme to further improve the quality of virtual view images.Experimental results on multiple datasets show that the proposed algorithm can significantly improve the quality of virtual views images,and ablation studys also demonstrate the effectiveness of each module of the algorithm.