Research On VR Systems Based On Ultra High Resolution Panoramic Video

In recent years,with the continuous development of virtual reality technology,people's audio-visual experience has been greatly enriched.VR panoramic video is currently the most popular VR service,being widely used in many scenes,thus it gains great attention from industry and academia.Compared to traditional video,panoramic video contains omnidirectional visual information and thus requires higher resolution.The ultra-high resolution panoramic video(16K or higher)contains more details and supports more playback modes,allowing users to zoom in and out of the region of interest,capable of providing users with more information.However,such a large resolution not only poses a huge challenge to the transmission bandwidth and terminal decoding equipment,but also introduces new requirements for video presentation and rendering method.In this paper,the ultra-high resolution panoramic video based VR systems with 3DOF and 3DOF+ degrees of freedom are studied.A tile-based viewpoint adaptive streaming method is proposed to improve bandwidth efficiency and reduce the actual processed video resolution,successfully solves the problems of transmission,decoding and rendering for ultra high resolution panoramic video.In the 3DOF VR system,this paper proposes four VR magnifier modes,which can enlarge the video region of the user's viewpoint to the circular area in the center of the viewport,while the other area of the viewport remains the same,thus allowing the user to zoom in the Ro I to view details clearly during playback process.Use 64 K ultra-high resolution panoramic video to ensure sharpness when zooming in.For the transmission and playback of ultra-high resolution panoramic video,the panoramic video is first evenly divided into 1152 tiles of 1280×1280 resolution,which are then encoded with MCTS method;during the playback process,tiles overleaped with Ro I are downloaded and stitched together as a single stream,which can be decoded by merely a single decoder.Tiles are used to render the circular magnifier area,so the terminal also needs to download the entire panoramic video of low quality to render the other area.Experiments show that our tile-based streaming scheme saves more than 96.17% of bandwidth as well as greatly reduce the actual processed video resolution compared to streaming the full 64 K resolution video.The four VR magnifier modes provide users with more interacting ways and more information,and significantly reduce the user's dizziness when zooming in.In order to provide users with more degrees of freedom,this paper further studies the 3DOF+ VR system based on ultra-high resolution panoramic video.The system allows users to move freely within the virtual sphere,viewing the panoramic video from any arbitrary position and angle.Users can move closer to the Ro I to zoom in the video protion,so that they can see more details.This paper proposes a layered video scheme consisting of a panoramic video's four different resolutions(8K/16K/32K/64K),among which the 16K/32K/64 K are used as HD layers and devided into tiles in different tile scheme.During playback,the terminal would select the most suitable layer to display according to user's position and orientation.To maintain sharpness,Players would switch to a higher video resolution as the magnification increases.The layeyed video scheme can gurentee the video resolution in FOV under the limit of decoder and make it smoother when switching between layers.We also propose ray cluster method and block-based rendering method to solve tiles' calculation and rendering problems.Experimental results show that our system saves more than 95.59% bandwidth compared with transmitting the whole 64 K video.In 3DOF+ mode,users enjoy more degrees of freedom and more information,have a stronger sense of immersion and reality.