Research On Video Retargeting Quality Assessment Based On Reverse Reconstructed Grid And Spatial-temporal Salience Classification

With the rapid development of multimedia display equipment and explosive growth of video data,it is necessary to adjust the spatial resolution or aspect ratio of video to make the video content adapt to the display requirements of different devices.Video retargeting algorithm based content aims to ensure the reconstruction quality of important content and keep the video playing smoothly as much as possible in the process of video zooming.However,the existed video retargeting algorithm is difficult to achieve ideal performance on complex video content.The methods more or less introduces the spatial and temporal distortion of video.In order to improve the performance of video retargeting method and choose the appropriate retargeting algorithm for different video applications,it is necessary to evaluate the quality of retargeting video effectively.The existing Video Retargeting Quality Assessment(VRQA)methods are mainly divided into subjective assessment and objective assessment.On the one hand,video retargeting quality subjective evaluation methods are usually looking for a small number of viewers for simple visual comparison,and the experimental results often not valid and convincing.So far,there is no independent and public subjective evaluation database of video retargeting.Due to the subjective database is a necessary benchmark to measure the performance of objective evaluation algorithm,and also the key to improve and enhance video retargeting methods.Therefore,it is necessary to research on subjective database of retargeting video quality.On the other hand,the existing Video Retargeting Quality Objective Assessment(VRQOA)method generally adopts the way of unified video size,which enlarge the frame of retargeted video to the size of the original video frame according to the SIFT method.The ways can solve the problem of inconsistent resolution of before and after retargeting,but it is easy to introduce distortion and affects the evaluation performance.Meanwhile the evaluation index of these algorithms set fixed weights to distortion index according to the human experience simply.It makes difficult to apply to quality evaluation on different content of retargeting video and lead to the evalution result difficult to get in line with the human visual perception finally.In order to solve the above problems,this paper make a study on the subjective and objective quality evaluation of video retargeting.The research results are as follows:This paper proposed and establish a video retargeting quality subjective assessment database to measure the effectiveness of the video retargeting quality objective evaluation algorithm.In this database,we invite 43 subjects randomly to evaluate the subjective quality of retargeted video using a double stimulus method.The database contains 28 original videos of various content and forms(including 20 videos which the spatial resolution at 352× 288,4 videos which resolution at 1280×720 and 4 videos which resolution at 1920×1080)and 840 test videos obtained by 6 retargeting methods.This paper proposed a video retargeted quality objective assessment algorithm based on spatial-temporal salience classification and adaptive fusion.We constructed a video classification model based on the spatial and temporal salience information.Then adaptively allocate different fusion weights to the evaluation indexes according to the characteristics of different kinds of video,and finally gets the final evaluation results by fusion.Specifically,we fusion Perceptual Geometric Distortion(PGD),Edge Group Similarity(EGS),Object Temporal Distortion(OTD)and Temporal Continuity similarity Distortion(TCD)to evaluate the quality of retargeted video.In particular,we propose TCD and OTD indexes to evaluate the temporal continuity between adjacent frames and the temporal distortion in important areas of retargeted video respectively.The experimental results show that compared with the existing evaluation algorithms,the proposed algorithm has better performance.The Kendall correlation coefficient between the proposed algorithm and the subjective database is 0.47.The proposed algorithm maintains good stability for different categories of videos.We also proposed a video retargeted quality objective quality assessment algorithm based on reverse reconstruction grid.First,this algorithm divide the retargeting video frame into uniform mesh.Then map the mesh vertices to the original video frame by reverse matching method,so as to generate the reverse construction mesh.Then,the methods using the energy index of video retargeting based on grid distortion(Warping)to measure the spatial distortion and temporal continuity of reconstructed mesh to evaluate the objective quality of retargeted video.It cans effectively solves the problem of video size mismatch caused by retargeting.The experimental results show that the Kendall correlation coefficient between the evaluation results of the proposed algorithm and the subjective database arrived 0.50.Compared with the existing algorithms,the proposed algorithm can evaluate the quality of retargeted video more effectively.And it has a higher degree of agreement with human perception,and lower processing time.