Research On The Rendering Techniques Of Panoramic Video

With the rapid development of virtual reality,panoramic video has become the most popular application form in virtual reality on account of its rich interestingness and immersion.Nowadays,panoramic video has been used in many fields,such as sport events living,travel adventure and video games,which makes it a strong demand for people to watch these videos online.However,as panoramic video records the scene from every direction,the resolution and data size of panoramic video are huge,which bring great challenges to the network and the video playback client.On the one hand,the streaming of panoramic video requires high bandwidth;on the other hand,decoding and rendering high resolution video also require high processing abilities of the client.As a result,the streaming and rendering are the most important factors which affect the viewing experience of the panoramic video.In this thesis,the research on the streaming and rendering techniques of panoramic video will be conducted.Among the processing steps of panoramic video,the mapping of panoramic video from the spherical domain to the plane domain plays a decisive role,and the choice of the projection format will have affects on the performance of the encoding and decoding of panoramic video and the way in which the panoramic video is streamed and rendered.Therefore,the mapping process of ERP(Equirectangular Projection)and CMP(Cube Map Projection),which are identified as the standard mapping projection format of panoramic video by OMAF,are analyzed in detail.The rendering methods of the panoramic video with corresponding projection formats are proposed and a rendering platform based on HTC Vive which renders the panoramic video using the aforementioned methods is built.However,both the projection formats have its own drawbacks when rendering panoramic video: a lot of redundant pixels exist in the region near the poles in the video frame of ERP format,which would become the bottleneck,and video frame in CMP format have seams on the edge of the cube,which would reduce the immersion of the video.To avoid such problems,a CPP(Craster Parabolic Projection)based rendering method of panoramic video is proposed.The experiment result shows that the proposed method has some advantages in comparison to the ERP and CMP based rendering methods.Subject to the current network status and the processing abilities of the client,streaming the full panoramic video frame would not only waste lots of bandwidth,but also become very difficult for the client to handle.Hence,the viewport-dependent streaming schemes are the mainstream streaming methods.Two viewport-dependent streaming methods,namely Tile-based and TSP(Truncated Square Pyramid)based streaming schemes,are proposed in this thesis.Combining the proposed streaming schemes with MPEG-DASH,two online panoramic video rendering system are built.Adaptive bitrate selection algorithm,bandwidth prediction algorithm,viewport prediction algorithm,playback speed control algorithm and TSP-based rendering methods are researched and implemented in the systems.In order to evaluate the clearness of video seen from the range of the user viewport,the average bitrates of pixels in the viewport is proposed as a new metric.Then the relationship between the average bitrates of pixels in the viewport and the prediction accuracy of the viewport,in the proposed two online panoramic video rendering systems as well as the ERP-based full viewport rendering system,are analyzed in detail.In the end,subjective experiments on the viewing experiences of user watching panoramic videos using three aforementioned online rendering systems under different prediction accuracy are designed and conducted.The results show that the proposed systems can provide good viewing experiences under limited bandwidth situations.