| Because free-viewpoint roaming system shows unique advantages in immersion and interactivity,it is widely used in flight simulation,virtual tourism and other fields.Among them,the free-viewpoint roaming system based on panoramic video has become one of the hotspots in recent years,because it needn't reconstruct three-dimensional models and can provide users with a full range of roaming experience.However,simply using images in panoramic video to generate virtual viewpoint images can not guarantee the quality of the generated virtual viewpoint images,which seriously affects user experience.This paper studies real-time free-viewpoint roaming systems based on panoramic video,and focuses on the problem of selecting reasonable reference images on the fly to generate a virtual viewpoint image in the process of free-viewpoint roaming.From the perspective of plenoptic function,we analyze the difficulties of selecting reasonable images on the fly from a collection containing a large amount of reference images,i.e.,the conflict between the computational overhead required to select reasonable images and the real-time performance of the system when the 3D information of the scene itself is unknown.In order to solve the contradiction,this paper proposed a precomputation-based reference image selection method,i.e.,moving the time-consuming reference image selection process to the precomputation stage,which ensures the quality of the selected reference images on the one hand,and the real-time performance of the system on the other hand.The main difficulty of selecting reference images in the precomputation stage is that the virtual viewpoints are generated by the user during the roaming process through interaction,while the information of the virtual viewpoints is unavailable during the precomputation stage.To solve this issue,this paper is based on an important fact that if two virtual viewpoints are similar enough in terms of their positions and view directions,their corresponding reference image selection strategies should also be similar.Based on this,this paper pre-set several viewpoints and view directions in the scene,and then precomputes their reference image selection strategies.In the real-time roaming stage,the corresponding reference image selection strategy can be obtained by simply making a nearest neighbor query to the precomputed results based on the position and the view direction of the virtual viewpoint,thus transforming the complex reference image selection computation into a time-saving query operation.Specifically,the main contributions of this paper are as follows:Firstly,a precomputation method and framework for reference image selection is proposed and implemented.In order to achieve a more reasonable reference image selection while meeting the high efficiency requirements of the system,a precomputation-based reference image selection method is proposed in this paper.The precomputation method in this paper requires pre-setting several viewpoints and view directions.To address the problem of high computational effort and redundant computation caused by uniformly and densely pre-setting the viewpoints and viewpoints in the roaming space,this paper proposes an adaptive viewpoint and view direction pre-setting method based on strategy similarity,which ensures their sufficiency while avoiding redundant computation.Secondly,from the perspective of the plenoptic function,a more reasonable reference image selection method and the optimal selection formula are analyzed,and a specific solution is proposed and implemented.In the specific selection method,for the difficulties brought by the unknown scene to select reference images more reasonably,this paper approximates the visible scene information under the virtual viewpoint to effectively select the subset of reference images or selection scheme that can provide more information to the virtual viewpoint.Thirdly,a suboptimal reference image selection scheme is proposed.With the precomputation method in this paper,it is easy to obtain the top K reference image options for the virtual viewpoint.Based on this,in order to improve the rendering efficiency of the system while ensuring the rendering effect,this paper makes appropriate trade-offs between the different options in terms of rendering efficiency and effect at runtime,and choose the compromise solution that can make the system achieve a better balance between rendering effect and efficiency.Finally,a reference image caching mechanism is designed and implemented on the panoramic video free-viewpoint roaming system in this paper,i.e.,the cached reference images are reused to reduce the number of reference images that need to be loaded in the rendering process for virtual viewpoints and to improve the rendering efficiency of the system.The experimental results show that the rendering effect can be effectively improved under the precomputation-based reference image selection method;with the reference image caching and sub-optimal reference image selection scheme,rendering efficiency can be improved while ensuring the rendering effect.The effectiveness of the adaptive method is verified by comparing the number and effect of pre-set view directions generated by the adaptive pre-setting method and the uniformly and densely pre-setting method;The effectiveness of the method proposed in this paper to approximate the visible scene under the virtual viewpoint is approved by comparing the reconstructed virtual viewpoint image with the ground truth image;finally,the effectiveness of the precomputation-based method in this paper is verified by comparing the rendering effect of the distance-based reference image selection algorithm,and the effectiveness of the design of the trade-off function is approved by the degree of change in the rendering effect of the system before and after the trade-off operation. |
PDF Full Text Request