Image Retargeting And Quality Assessment

With the popularity of mobile devices and the development of stereoscopic display technology,people are no longer satisfied with the visual availability,but are more pur-suing the visual enjoyment.Image retargeting is widely concerned and studied because it can adaptively adjust multimedia content to meet the needs of different terminals.In order to meet the hard requirements of the device display and the soft requirements of the user's visual experience,quality assessment is often taken into account by many ma-ture industry chains.Therefore,this article discusses the complementary relationship between image retargeting and quality assessment.On the one hand,we study the qual-ity assessment of image retargeting;on the other hand,we study the image retargeting technology under the guidance of quality assessment.Research on stereoscopic image retargeting is relatively scarce,and there is also no public dataset.In order to fill this gap,we establish the first subjective dataset of stereoscopic retargeted images named Stereoscopic Image Retargeting Dataset(SIRD).It involves four typical stereoscopic retargeting methods including single operator and multi-operator.It includes four stereoscopic perception evaluation dimensions,includ-ing image quality,visual comfort,depth quality and the overall quality.And it is with strong representativeness and high confidence.In order to evaluate the safety of users when viewing stereoscopic retargeted im-ages,and meanwhile to simulate the binocular perception mechanism of the human vi-sual system,we design the first visual comfort assessment model,named Hierarchical Visual Comfort Assessment(Hi-VCA),based on hybrid distortion aggregation.In view of the structural distortion,information loss,binocular incongruity,and semantic distor-tion that often occur in stereoscopic image retargeting,we design four measurements to comprehensively describe the impact of retargeting on comfort,respectively.Especially in the binocular incongruity measurement,we consider several abnormal binocular per-ception mechanisms,including visual comfort zone,binocular rivalry,window conflict,and accommodation-vergence conflict to more thoroughly analyze the factors that in-duce visual discomfort.Finally,the experimental results show that the Hi-VCA model has superior visual comfort assessment ability not only for stereoscopic retargeted im-ages,but also for ordinary stereoscopic images,which is extremely competitive and practical among related models.In order to solve the problem of low data volume and high labeling cost in quality assessment,we propose a quality assessment scheme based on active learning.For the traditional quality assessment models,we have designed a sampling strategy based on clustering.For the quality assessment models based on deep learning,we first design a double-path stereoscopic image quality assessment network,and then propose a sam-pling strategy based on perceptual similarity.Experimental results demonstrate that,although the labeling cost is reduced by about half,it can still achieve the performance comparable to that of using all sample points.In order to realize the inverse optimization of image retargeting by quality assess-ment,we design a multi-operator image retargeting framework based on reinforcement learning and quality perception,named Semantics and Aesthetics aware Multi-operator Image Retargeting(SAMIR).First,we model the multi-operator retargeting task as a Markov decision-making process.Then,semantics and aesthetics aware measurement is proposed as reward function of reinforcement learning,on the one hand to ensure the correct transmission of image information,on the other hand to ensure the visual quality of the image.Finally,without the need for any annotation data and with low complexity,adaptive selection of operators in multiple steps is achieved.The SAMIR framework also provides users with a certain creative space.By designing the action space,and adjusting the retargeting direction and proportion,more diverse effects can be realized according to the needs.