Panoramic Video Super-resolution For Immersive Experience

Panoramic video covers content in the range of 360°×180°,providing users with an immersive experience.It has become the main content source of virtual reality video applications.Because of the high resolution and large volume of panoramic video,it brings great pressure to the network if directly used for transmission.Considering that the viewport is limited and in order to improve the utilization of network bandwidth,the existing transmission scheme is mainly viewport-based transmission.This scheme reduces the bandwidth occupied during transmission,but brings delay and affects the immersive experience when audiences switch the viewport.This paper presents a panoramic super-resolution algorithm to relieve the uncomfortable effect on immersive experience.The main idea is as follows: when the high-resolution content corresponding to the viewport requested by the user has not yet arrived,the image or video reconstructed by the algorithm would be provided to the user first.The generated image or video can satisfy the user's immersive experience and avoid dizziness and other problems caused by delay.The main research work is as follows:(1)Based on the viewport transmission scheme,this paper proposes a panoramic image super-resolution algorithm.It first predicts the initial viewport and transfers its high-resolution version to the client.For the rest of the image,it transfers the corresponding low-resolution content.At the client,it first calculates the relationship between the low-resolution image and the high-resolution viewport.Then,by texture fusion and super-resolution algorithm,it reconstructs the high-resolution version of the image outside the viewport.The experimental results show that compared with other initial viewports,the correlation between the predicted initial viewport and the surrounding area is higher,which helps to reconstruct more accurate pixel values.On the other hand,for simple texture and complex texture,the algorithm achieves better visual performance and higher objective index values than other algorithms.(2)Using non-local feature fusion and gradient guidance,this paper proposes a super-resolution algorithm for part of the panoramic image.It uses multi-scale non-local operations to match and fuse features between image blocks.In addition,in order to retain the overall structural characteristics of the low-resolution region,a gradient-guided branch is introduced.Finally,the reconstruction results of the two branches are constrained simultaneously.The experimental results show that this algorithm can reconstruct more realistic texture details and edges closer to the real image.The objective index values of reconstructed images are higher than other contrast methods.(3)This paper introduces inter-frame correlation and proposes a spatio-temporal super-resolution algorithm for panoramic video.In order to reduce the amount of data transmitted,this paper proposes to transmit low-resolution and low-frame-rate version for areas outside the viewport.For areas within the viewport,it transmits the high resolution,high frame rate version.Firstly,to obtain the feature map of the intermediate frame,the high-frequency information of the high-resolution viewport is extracted by octave convolution and fused with the features of the low-resolution region.Then the motion relationship between each frame is captured by a long short-term memory network.Finally,the high-resolution image of each frame is reconstructed by the network.Experimental results show that,compared with other methods,the proposed algorithm can generate more coherent intermediate frames.The reconstructed high-resolution frames have better objective and subjective quality.