Study On The Objective Quality Assessment Metrics Of Virtual View Synthesis

Recently,the 3D-related technologies,such as virtual reality,multi-view and free-viewpoint videos,have attracted much attention for the widespread application in many areas like remote education,medical applications and entertainment,etc.Virtual view synthesis technology is the key to realize these applications,and the visual quality of synthesized views directly determines whether this technology can be successfully applied.Therefore,studying the Quality Assessment(QA)of virtual view synthesis is of great significance.Based on analysis of the research status,this thesis studies the key problems in this direction and proposes the corresponding solutions,including QA for the synthesized images degraded by the distortions introduced during the DIBR process,QA for the whole view synthesis process and QA for virtual synthesized videos.The research achievements of this thesis can be used to guide the design of view synthesis algorithms and the coding of multi-view texture and depth,which have important theoretical research significance and application value.The main work and contributions of this thesis are summarized as follows:(1)A quality metric for DIBR-synthesized images based on the measurement of local geometric distortions and global sharpness is proposed by exploring the effect of the distortions introduced during the DIBR process on the synthesized image quality.DIBR is the key technique in view synthesis,the limitations of which will mainly introduce local geometric distortions,but also some global distortions to the synthesized images.Therefore,it is necessary to evaluate from both local and global aspects.However,the existing DIBR-synthesized quality metrics mainly evaluate from the local or global aspect and they cannot accurately locate the local distorted regions,which cause their limited performance.Based on these,this paper proposes a metric based on both local and global quality measurements.For the local measurement,we propose a SIFT-flow based location method and a distortion quantification method.For the global measurement,considering that rendering is also affected by image blur,a reblurring-based method is proposed to quantify the global sharpness.Finally,the local and global scores are integrated to produce the final quality score.Extensive experiments on several public view synthesis databases demonstrate the superiority of the proposed method.Moreover,the proposed method is also proved to perform the best for the performance ranking of DIBR algorithms.(2)A DIBR-synthesized image quality metric based on the measurement of instance degradation and global distortions is proposed by further exploring the characteristics of the HVS when perceiving images.Low-level features are extracted to describe the local distortions in DIBR-synthesized images in the last work.However,the HVS tends to perceive images from a higher level,especially from the salient objects(instances),and the local distortions in synthesized images cause the instance degradation.Inspired by this,we propose to better describe the local distortions via the quantification of instance degradation.For the global quality,the blurring effect and geometric deformation are both evaluated based on the HVS characteristics,which is lacking in existing quality metrics.Finally,the local instance score and the global score are integrated to produce the final quality score of the synthesized image.Extensive comparison experiments conducted on the synthesized databases demonstrate the superiority of the proposed method,and also prove that the proposed method can be applied for the performance evaluation of DIBR algorithms.(3)A general-purpose full-reference(FR)quality metric for view synthesized images is proposed by exploring the influence mechanism of the distortions introduced by each stage of the whole virtual view synthesis process on image quality.All existing view synthesis quality metrics,including the above two works proposed in this thesis are specifically designed for the evaluation of the distortions introduced by only one stage of the view synthesis process.However,in practice,any stage of the view synthesis may introduce distortions to the synthesized view,and the distortions caused by different stages have quite different effects on the quality of the final synthesized views.Therefore,it is of great significance to study the general-purpose quality metric for the whole view synthesis process.Based on this,we study the effects of the distortions introduced by each stage and find that they all cause low-and mid-level structural degradations.Hence,we propose a general-purpose quality metric for view synthesis based on the evaluation of both low-and mid-level structural degradations.Among them,low-level structural degradation is represented by the statistics of both gradient intensity and orientation,and mid-level structure is described via the sparse coding.The experiments conducted on the view synthesis image databases demonstrate the effectiveness of the proposed method in evaluating the whole view synthesis process.(4)A general-purpose no-reference(NR)quality metric for synthesized views is proposed by further exploring the effect of the distortions introduced by each stage of the whole view synthesis process on the natural scene statistics of synthesized images.The metric proposed in work(3)is a general-purpose FR metric.However,the reference images of virtual synthesized images are usually difficult to get in the practical applications,which causes the big application limitation of the FR quality metric.By contrast,the NR quality metric has greater research value.Inspired by the general-purpose NR quality metrics based on natural scene statistics,we deeply analyze the common characteristics of the effects of the distortions caused by each stage on the statistical property of synthesized views,and find that the distortions destruct both edge statistics and textural naturalness around image edges.Based on this,we propose a quality metric for synthesized views based on edge statistics and textural naturalness.Besides,with the consideration of the facts that the Difference-of-Gaussian(DoG)representation has the property of simulating the characteristics of the HVS,sensitivity to distortions and effectively representing the image edge and texture information,we propose to extract the edge statistic and textural naturalness features from the DoG images.Experiments indicate the better performance and generalization ability of the proposed method than both traditional and view synthesis quality metrics in evaluating the distortions in each stage.(5)A NR quality metric for synthesized videos is proposed by exploring the characteristics of the temporal distortions in virtual view synthesized videos.In practical applications,the multi-view and free-viewpoint applications mostly appear in the form of videos.Therefore,studying the QA of synthesized videos has broad application prospects.Both the image quality metrics specific for spatial distortion measurement and the video quality metrics designed for traditional distortion measurement cannot effectively evaluate the quality of synthesized videos.The main reason is that the distortions in synthesized videos are mainly the local temporal flickering distortions.Inspired by this,we propose a flickering region location method and a flickering distortion quantification method in the singular value decomposition domain.The experiments on two synthesized video databases demonstrate the accuracy of the proposed flickering region localization method,the superiority of the proposed method compared with existing quality metrics,and the highest consistency between the performance rankings of the view synthesis algorithms achieved by the proposed method and those achieved by the subjective results.This thesis consists of 36 images,44 tables and 208 references.