Research On Image-based Free Viewpoint Synthesis

Image-based free viewpoint synthesis,which is also known as image-based rendering(IBR),refers to directly render the novel view from the unshot viewpoint using a group of real world images taken in advance through the computer vision,so as to realize the interactive 3D virtual navigation.Compared with traditional model-based methods,the IBR directly render high-quality views without complicated manual adjustment and simulation.IBR methods can be divided into two types according to whether they need geometric priors.The IBR methods based on geometric priors usually use the Multi-view Stereo(MVS)algorithm to calculate the geometric information,and then use it to guide the view synthesis via input images.The other part of the IBR methods can render the novel views directly from the input images without geometric priors.The IBR methods based on geometric priors performs well when geometric information is rich and accurate,but the artifacts will appear when the geometry is missing or wrong,which will reduce the view quality.Currently,some IBR approaches based on geometric priors have yielded impressive results on the indoor scenes with dense photo capture due to sufficient reconstruction.However,the outdoor scenes often contain a large number of vegetation,sky and other low-textured areas without features.Even the most advanced MVS method is still unable to reconstruct complete and accurate scene geometry in these areas.Even the most advanced MVS algorithm cannot reconstruct a complete and accurate scene geometry in these areas.As a result,there is still no effective free viewpoint synthesis method for outdoor scenes.For alleviating this problem,this paper proposes a free viewpoint synthesis method based on depth fusion,which improves the outdoor scene geometry by fusing the MVS depth map and the monocular depth map estimated by deep learning,and then improves the final view synthesis quality.On the GTAV large outdoor dataset,the RMSE(Root Mean Square Error)error of our fused depth is reduced by 9%compared with the original MVS depth,and the RMSE error is reduced by 7%compared with the refined depth by the method using MVS depth propagation.Compared with the existing IBR methods,our method effectively improves the view synthesis quality when using the same MVS depth.An important research direction without geometric priors is light field.The traditional light field rendering method requires dense and regular image capture,which is difficult to practically apply.Recent years,with the development of deep learning technology,researchers have discovered that the neural network can be used to fit the light sampling,and implicitly encoding the light field from input images to render new views.Compared with the traditional light field,the Neural Reflectance Field(NeRF)method can be used for handheld capture and few input images,which greatly expands the application.NeRF needs to sample each ray when fitting the light field,and its view quality is proportional to the number of effective samples.However,the increase number the of samples means more calculations and reduced efficiency.The current NeRF method still has the problems of long training time and slow rendering of views.Therefore,this paper proposes a joint sampling-based NeRF,which can make the coarse network and fine network share uniform sampling results,thereby accelerating the network training and view synthesis by reducing unnecessary light sampling.The experiments demonstrate that,in the case of the similar view quality,compared with the baseline method,the proposed method can reduce the training time by 20%and improve the view synthesis efficiency by 25%.