Research On Multimedia Quality Of Experience Based On Perceptual Properties

With the rapid development of information technology,multimedia has gradually become an important media for people to express and communicate,and the multimedia information has also explosively increased.Under this background,related multimedia processing techniques have also become research hotspots.The ultimate receiver of multimedia information is usually human,thus studying and utilizing human perceptual properties can improve relevant multimedia processing techniques effectively.By studying human perceptual properties,this thesis explores several techniques related to multimedia quality of experience(QoE),especially QoE assessment.Human perception of multimedia information is a bottom-up process of continuously processing the low-level perceptual signals and extracting high-level semantic information,and the whole information perception is conceptualized as several hierarchical stages: low-,middle-and high-level perception.Low-level perception mainly acquires basic features such as brightness,color and orientation from the input perceptual signals.Middle-level perception mainly obtains more advanced middle-level features such as image structures and contours by analyzing and integrating the basic features.High-level perception mainly obtains content and semantic information such as objects and scenes by further analyzing the middle-level features.This thesis explores the low-,middle-and high-level perception of multimedia information,and utilizes the perceptual properties of humans in various stages to build multimedia QoE assessment models,which can help to provide a better QoE.Firstly,humans utilize the attention mechanism in the low-level perception to select the key information for more advanced processing.This thesis models such attention in the low-level perception and utilizes visual attention to assist visual quality assessment.Specifically,this thesis incorporates more factors that affect visual attention to improve the current visual attention models.To assist visual quality assessment,this thesis also utilizes visual attention to assign different weights to the contents with different importance.Traditional visual attention models generally make full use of visual features such as intensity,color and orientation to detect areas which are significantly different from the surroundings.These models perform well in scenes which can be well represented by these visual features,but they are not so efficient in some other scenes.Thus it is necessary to incorporate some other factors that affect visual attention to improve the current visual attention models.Audio information is generally overlooked by current visual attention modelling research,however,audio information has some impacts on visual attention,so this thesis tries to model visual attention using both audio and visual information and proposes an audio-visual attention model.Through audio-visual correlation analysis,this thesis locates the moving-sounding objects as the audio attention maps,which are then combined with the traditional visual attention maps to obtain the final audio-visual attention maps.Face information is another important factor that has a significant impact on visual attention.Although some visual attention models incorporate face detection,the way they incorporate face information is not comprehensive.So this thesis analyses the visual attention distribution on faces systematically and proposes visual attention model specifically for face images.Based on face detection and landmark localization,this thesis extracts a series of facial features,which are then combined with the traditional saliency maps to obtain the predicted face visual attention maps.Results of related eye-tracking experiments show that traditional visual attention models have been significantly improved after incorporating the above factors.As an underlying mechanism of visual perception,the visual attention mechanism can assist higher-level visual signal processing,including visual quality assessment.Therefore,this thesis also draws on the visual attention mechanism by assigning more weights to the salient regions and assigning less weights to the non-salient regions to assist visual quality assessment.Secondly,the integration of the basic features in the middle-level perception can be affected by the quality degradation.This thesis utilizes such distortion perception properties in the middle-level perception to build QoE assessment models.Specifically,this thesis introduces a concept of the most distorted image(MDI),then builds a blind image quality assessment(IQA)framework based on the MDI,and proposes a series of distortion-specific and general-purpose blind IQA(BIQA)algorithms based on this framework.Traditional quality assessment algorithms generally follow a common framework,in which the deviation of the target image from the high quality image is measured to predict the image quality.Motivated by this framework,this thesis introduces the concept of MDI.Different from high quality reference image,the MDI is generated from the distorted image and it describes the conditions of the distorted image under severer distortions.This thesis takes the MDI as a base point to describe the worst image quality,and estimates the image quality by measuring the deviation of the distorted image from the MDI.Based on the MDI-based BIQA framework described above,this thesis proposes MDI-based blockiness,sharpness and noiseness estimators,and then integrates these distortion-specific estimators into a general-purpose MDI-based BIQA method via a 2-stage quality regression after distortion identification framework.This thesis further improves the above algorithms,and introduces multilevel distortion aggravated images(MDAI)by adding several types and levels of distortions to the distorted image,and then compares the similarities between the distorted image and the MDAI.More similar to a specific distortion aggravated image indicates that the distorted image's quality is closer to this image,and the proposed MDAI-based BIQA method is integrated from a series of similarity scores between the distorted image and the MDAI.Experimental results on mainstream IQA databases verify the effectiveness of the proposed algorithms.In addition,the proposed algorithms are more stable than mainstream BIQA algorithms,since this thesis estimates image quality by comparing images and the influence of image content is significantly reduced in such comparison process.Finally,humans obtain content and semantic information in the high-level perception,while the perceived quality highly relies on high-level content and semantic information.This thesis utilizes such content perception properties in the high-level perception to extend the application scope of multimedia QoE assessment.Specifically,this thesis investigates cross-content-source visual quality assessment,and proposes cross-content-source IQA algorithms.With the rapid development of applications such as remote terminals,cloud computing and live games,the content sources in visual communication systems have been greatly extended.Different from photographic natural scene images(NSIs)which are captured from real-world scenes,computer graphic images(CGIs)and screen content images(SCIs)are mostly generated artificially by computers,and they have substantially different characteristics from NSIs.Therefore,it is necessary for us to study the applicability of IQA in various content sources,and to extend the current IQA algorithms to various content sources.On one hand,this thesis conducts subjective cross-content-source IQA studies and proposes a unified content-source adaptive(UCA)BIQA algorithm which is applicable to NSIs,CGIs and SCIs.Based on a novel perceptually motivated content source adaptive multi-scale weighting strategy proposed by us,UCA algorithm can extract and integrate corner and edge features at multiple scales adaptively according to the image content characteristics,thus can better simulate the perceptual properties of the human eyes when viewing different contents.On the other hand,this thesis analyses the relationship between visual attention and visual quality assessment,and proposes a cross-content-source saliency-induced reduced-reference(SIRR)IQA algorithm.The SIRR algorithm is inspired by two points: first,the quality degradation can cause the change of low-level image features,which will affect the saliency detection;second,the saliency detection is actually a data reduction operation.Therefore this thesis estimates image quality by measuring the differences between saliency maps of two images,while the saliency map is described by a binary image descriptor which occupies only a small amount of data.Experimental results on the constructed and existing IQA databases verify the effectiveness of the proposed algorithms.Compared with traditional IQA measures,the proposed algorithms show stronger robustness across both content and content source variations.