| Possessing ultra high encoding bitrate makes the user experience of panoramic video hard to guarantee in limited radio resources.Therefore,a tile-based scheme of differential bitrate in the field of view(FoV)and nonfield of view is proposed.In this scheme,viewpoint prediction and bitrate adaptive algorithm are two significant factors that affect user experience,but there’s room for improvement in the accuracy of existing viewpoint prediction algorithms,and traditional bitrate adaptive algorithms based on fixed rules appear less intelligent in the time-varying network environment and complex decision-making problem.To tackle these challenges,the research on viewpoint prediction and intelligent transmission optimization for panoramic video is carried out.The main contents and contributions are summarized as follows:First,to improve the accuracy of viewpoint prediction,a method that integrates user trajectory and video content is proposed,which comprehensively considers the short-term inertia of user’s head rotation and the impact of video content on viewing behavior,as well as the excellent performance of LSTM in time series prediction.The video content mainly includes saliency and dynamics features,and they are fused to calculate the barycentric coordinate in this paper.Specifically,input the viewpoint coordinates of several frames into the LSTM network to predict a temporary viewpoint coordinate,and then connect it with the barycentric coordinate of the fused feature map with the full connection layer to generate the fina’ predicted viewpoint coordinate.The simulation results show that the prediction accuracy is improved by about 15%compared with linear regression and the LSTM prediction method using only the viewing trajectory.Finally,meta-learning is introduced to further solve the problem of the lack of generalization ability in new tasks under few viewpoint data.The simulation results demonstrate that the model with meta-learning can achieve stronger rapid adaptability at new tasks than the basic LSTM model.Second,traditional bitrate adaptive algorithms that based on fixed rules appear less intelligent in the complicated multivariable joint optimization problem under time-varying network bandwidth,player buffer,etc.In this paper,we first redefine a new QoE model taking blackarea ratio into account,which is brought by viewpoint prediction error.Then,a bitrate and redundance ratio adaptive algorithm based on deep reinforcement learning(DRL)is proposed.Specifically,the adaptive transmission problem is modeled by Markov decision process,where the state includes the bitrate and redundance ratio of the last segment,predicted bandwidth,player buffer occupancy,data volume corresponding to each optional action,etc.The action is the combination of next segment bitrate and redundance ratio,and the reward is defined as the total QoE of the whole playback process.Simulation results verify the effectiveness of DRL algorithm,and we further explore the impact of parameters such as the capcity of replay memery and discount factor on the performance of DRL network.Finally,compared with existing baselines such as BB,RB and MPC,our proposal achieves competitive performance in both the total QoE(17%～32%improvement)and the joint optimization of sub-indicators.In conclusion,this paper proposes a viewpoint prediction algorithm based on LSTM and meta-learning,and a bitrate and redundance ratio adaptive algorithm based on deep reinforcement learning.Simulation results indicates that the former is capable of improving the accuracy and generalization of viewpoint prediction,and the latter can improve the intelligence of decision-making.Both of them can provide guidance for the optimization of panoramic video user experience under limited radio resources. |
