Perceptual Quality Assessment Of Omnidirectional Images

Owing to the development of virtual reality and 5G technology,omnidirectional contents have gradually entered the lives of ordinary people in recent years.360-degree images are favored by manufacturers and consumers since they can provide users with the 360°×180° spherical scene and strong immersive feelings.However,due to the lim-itations of the shooting system,transmission bandwidth,display equipment,as well as the inevitable distortion introduced during the acquisition,compression,transmission,reconstruction and display,the perceptual quality of omnidirectional images is far from satisfactory.Therefore,in-depth researches on the perceptual quality of omnidirectional images can effectively guide the optimization of acquisition equipments and image pro-cessing systems,and monitor the quality of omnidirectional contents to maximize the user experience.Different from traditional 2D images and videos,omnidirectional contents can pro-vide consumers with freely changeable viewports and a larger field of view.However,the projection deformation and field of view(FoV)range in 2D omnidirectional im-ages make the objective quality assessment of omnidirectional images more challeng-ing.Current omnidirectional image quality assessment(IQA)models only modify the traditional IQA metrics and expand them into the omnidirectional versions,without too much consideration of the characteristics of omnidirectional contents and human vision system,leading to the low prediction accuracy.Moreover,there are few quality assess-ment works for stereoscopic omnidirectional images.Apart from the image quality,depth quality and visual comfort are factors affecting the user experience.First,we analyze the viewing process of omnidirectional images and propose the blind quality assessment model by building the spatial viewport graph.Second,we establish the first stereoscopic omnidirectional image quality assessment database to guide the design of stereoscopic omnidirectional IQA metrics and provide the benchmark for different met-rics comparison.Third,considering the human binocular vision and characteristics of stereoscopic omnidirectional images,we propose the full-reference stereoscopic omni-directional IQA and overall quality of experience(QoE)assessment model.(1)Inspired by the browsing process of omnidirectional images as well as the hu-man vision system,a Viewport oriented Graph Convolutional Network(VGCN)is pro-posed to evaluate the quality of omnidirectional images without the reference.VGCN is composed of the local and global branches to simulate the viewport information in-teraction and aggregation,and reconstruction in the hallucination,respectively.In the local branch,the high-probability visible viewports are selected to construct the spatial viewport graph,and the interaction between nodes is captured through graph convolu-tion.In the global branch,the deep bilinear convolutional neural network is used to measure the synthetic and authentic distortions.The effectiveness and generalization of VGCN are verified by the performance comparison on the public databases.(2)Since human beings are the final receivers of visual contents,it is necessary to collect real subjective experimental data to analyze the key factors affecting the per-ceptual quality of stereoscopic omnidirectional images.Thus,we establish the first Stereoscopic OmnidirectionaL Image quality assessment Database(SOLID).Six refer-ence images are captured by Facebook Surround 360 system.After pre-processing,276 distorted images are generated by adjusting different disparity parameters(zero dispar-ity,medium disparity,large disparity)and introducing different distortion types(JPEG compression,BPG compression)and distortion levels.We conduct the subjective ex-periments with head mounted displays to evaluate the image quality,depth quality,and overall QoE.The analysis of subjective experiments shows that image quality is mainly affected by compression parameters,depth quality is mainly affected by disparity pa-rameters,and overall QoE is mainly affected by the image quality.(3)Subjective experiments are unpractical in real-time applications due to the high cost and low efficiency.Therefore,to achieve automatic prediction of the per-ceptual quality,we analyze the characteristics of stereoscopic omnidirectional images and propose the full-reference Stereoscopic Omnidirectional Image Quality Evaluator(SOIQE)and the QoE assessment Multi-ViewPort based Model(MVPM).According to the binocular vision mechanism and the projection deformation of omnidirectional images,SOIQE contains the binocular rivalry module and the multi-view fusion mod-ule.In the binocular rivalry module,prior and likelihood probabilities are calculated via the predictive coding theory to obtain the rivalry dominance for left and right views.In the multi-view fusion module,content weight and location weight are utilized to fuse the qualities of different viewports.The structure of MVPM is similar to SOIQE,it improves the prediction accuracy for overall QoE by further incorporating the depth features.The experimental results on the self-built SOLID database and other public databases demonstrate the validity of the proposed SOIQE and MVPM.