Research On View Synthesis Technology Based On Multiplane Image Scene Representation

The study of novel view image synthesis is one of the current research topics in the field of computer vision.Multiplane image(MPI)constructs a camera-centered,depth-layered explicit representation of the 3D scene,which can effectively describe the geometry of the scene.Better synthetic perspective images can be obtained with high-quality MPI scene representation,but there are still artifacts and distortions that are difficult to eliminate.In order to resovle these problems,the following research has been conducted and completed:(1)A new MPI novel view synthesis algorithm based on image dense feature extraction is proposed,which explicitly models the dependency between convolutional feature channels in the network encoding stage and uses dynamic channel feature recalibration to optimize the ability of the encoder network representation.Dense feature extraction is implemented on the input view images,and geometry semantic features of the images are obtained from it.Experimental results show that the MPI scene representation inferred by the algorithm from the input view images can achieve an accurate description of the geometric semantics of the scene,thereby the quality of synthesized novel view image is improved.(2)A novel view synthesis algorithm based on MPI is proposed that makes use of feature connections across depth layers.The effective spatial features between multiple depth planes are captured by using 3D convolutional residual blocks,and the prediction ability of MPI depth plane occupied regions is improved,therefore,further the prediction accuracy of the geometric semantics of each depth plane is improved.Numerical experiments show that the algorithm can effectively eliminate artifacts and distortions in the synthesized novel view images in the view extrapolation and view interpolation tasks.When the horizontal baseline width of the reference view is doubled and the number of MPI depth planes is not increased,better numerical results are still obtained.(3)A new MPI novel view synthesis algorithm(Trans MPI)based on global feature modeling is proposed,and a self-attention mechanism is introduced on the basis of the network architecture in Chapter 4 to overcome the inductive bias of the convolutional network for global semantic information learning.The network of Trans MPI uses the obtained local features,combined with the Transformer encoder to achieve global feature representation modeling,therefore,the long-distance dependencies between features are established.Experimental results show that the inference quality of MPI scene representation and the quality of synthesized novel view images are further improved by utilizing the self-attention mechanism to learn global and local features between consecutive depth planes in Trans MPI.