Design And Implementation Of Free-viewpoint Roaming System Based On Panoramic Video

As a new form of media input,panoramic video is widely used in the scene roaming system due to its high degree of freedom and high immersion.However,the panoramic video can only view 360° scene information on a fixed path,and cannot view the scene from a free-viewpoint,so it cannot realize the free-roaming of the scene.To realize the free-roaming of the scene,Depth-Image-Based Rendering(DIBR)is usually used to synthesize images of any viewpoint in the scene by using known image information and depth information.However,there are cracks,overlaps,voids,artifacts,and efficiency in DIBR,affecting the quality of synthetic images and computational efficiency.To solve these problems,this paper designs and implements a high-quality interactive freeviewpoint roaming system with cube panoramic video and its corresponding depth as input.On the one hand,to improve the quality and computational efficiency of the synthesized image,this paper proposes an iterative reverse warping view synthesis algorithm based on depth information,which can use the parallel computing framework to synthesize high-quality virtual views in real-time.On the other hand,due to the large performance overhead of decoding panoramic video,to ensure the real-time operation efficiency of the system,this paper proposes a depthbased invisible information culling strategy,which can reduce the video content that needs to be decoded and the computational cost of the view synthesis algorithm in this paper.Specifically,the main work of this paper includes :· Firstly,this paper proposes an iterative view synthesis algorithm based on depth information assistance.The algorithm applies the idea of fixed-point iteration and uses the motion vector of pixels to construct the inverse warping relationship between the reference view and the virtual view,which can perform the parallel iterative calculation for each pixel in the virtual view.To speed up the convergence speed of iterative calculation,this paper uses the depth information of virtual viewpoint obtained by forwarding warping and inverse warping to the reference view to obtaining the position of the initial points of the iteration.It can effectively solve the problem of overlap and crack in virtual view by using fewer initial points and quickly synthesize high-quality virtual view.In addition,this paper also uses the view post-processing algorithm,including the ghost removal algorithm based on hole edge expansion and the hole filling algorithm based on depth and motion vector guidance to deal with the ghost and hole problems in virtual views,which further improves the quality of virtual views.· Secondly,this paper proposes an invisible information culling strategy based on depth.In this strategy,each face of cube panoramic video is further divided into smaller resolution Tile video,and the maximum depth and minimum depth information corresponding to each frame of Tile video are obtained by using the pre-calculation.During the realtime operation of the system,the view frustum bounding box of the scene information contained in the Tile view can be quickly constructed by the maximum depth and the minimum depth.Then,the visibility of the view frustum bounding box for the virtual viewpoint is quickly judged by the calculation method based on feature points,to without decoding the invisible Tile video in the panoramic video.This strategy can reduce the video content that needs to be decoded,improve the decoding efficiency of the system,and reduce the amount of calculation of the view synthesis algorithm in this paper,to improve the efficiency of the system.· Finally,a free-viewpoint roaming system based on panoramic video is designed and implemented.The system in this paper uses FFmpeg and Cuda to construct a multi-threaded real-time video hardware-decoding framework,which can quickly decode the required reference view.At the same time,Open GL is used to realize the view synthesis algorithm and view post-processing algorithm,which can synthesize and display virtual views in real-time.Besides,the system also provides users with real-time system interaction functions,including the transformation of the viewpoint position and the arbitrary rotation of the camera,allowing users to roam the scene freely in real-time,greatly improving the user's interactive experience.The experimental results show that the iterative view synthesis algorithm based on depth information assistance can obtain high-quality virtual views and has high computational efficiency.At the same time,the invisible information culling strategy based on the depth proposed in this paper can effectively reduce the time cost of video decoding and view synthesis algorithm,and improve the operating efficiency of the system.The operating efficiency of the system in this paper is more than 60 FPS,which can guarantee the real-time free-roaming scene for users.