Research On The Methods Of Compressed Multi-view Image Enhancement In 3D Scene

Efficient compression is a prerequisite for video communication,especially for Multi-view video plus depth(MVD)for future interactive video applications.Although the efficient compression of MVD brings possibilities for video transmission and storage,the compression distortion generated during the coding process is irreversible,which has a significant impact on the image presentation,human-computer interaction,and image rendering for end users.How to effectively improve the MVD quality of the end user without the original high-quality signal at the encoding side is a challenging problem to be solved urgently in the practical application of future interactive video.The loop filter in the traditional video codec can improve the video quality by using the spatial information of image,but it has not explored the interaction mechanism of MVD data.This leads to limited space for quality improvement,which hinders the future application of MVD.In view of the above challenges,this paper establishes the interaction mechanism of MVD data in multiple dimensions such as viewpoint domain,temporal domain,and spatial domain.Under the framework of asymmetric coding,we have conducted research from different perspectives such as signal filtering,crossentropy evaluation,and graph model evaluation.Some progress has been made,including:(1)A cross-view multi-lateral filtering scheme is proposed to enhance the distorted depth map via non-local candidates selected from current and neighboring viewpoints of different time-slots;a macro super pixel structure based on candidate regions is constructed,which describe the physical and semantic cross-relationships of the crossview,spatial and temporal priors;a multi-lateral filter model is constructed to describe the interaction between the above multi-modal priors.The contribution of different modal priors to the quality improvement of depth map is mined,and the noise interference in each modal prior is suppressed.The experimental results show that our proposed filter can obtain an average gain of 2.63 d B and 0.032 on the PSNR and SSIM,respectively.In subjective evaluation,the target contour of the depth image can be restored even under the conditions of high compression ratio and high distortion.We also verify the performance of our method through virtual viewpoint rendering and 3D modeling for practical interactive video applications.In these verifications,the artifacts on objects contours in the image can be effectively processed,and the discontinuous object surface in 3D modeling can be recovered well.(2)A multi-view optimized filter for depth image quality enhancement is proposed and the contribution mechanism of the distorted priors in filtering the depth image of current viewpoint is studied.The expression mode of the inner-and cross-view priors in the filter design is explored,which overcomes the problem that distortions of priors in traditional methods cannot contribute to the filter;the entropy correlation coefficient based on super-pixel is designed as the view consistency index,which effectively evaluates the contribution of cross-view prior information in the distorted MVD;the method of modeling the inner-and cross-view priors under the global optimization framework is researched,and an energy function with corresponding data accuracy and spatial smoothness is designed.The experimental results show that the proposed model outperforms state-of-the-art methods,where 3.036 d B and 0.0375 average gains on PSNR and SSIM metrics can be obtained,respectively.For subjective evaluation,object details and structure information are recovered in the compressed depth video.We also verify our method via several practical applications,including virtual view synthesis for smooth interaction and point cloud for 3D modeling for accuracy evaluation.In these verifications,the ringing and malposition artifacts on object contours are properly handled for interactive video,and discontinuous object surfaces are restored for 3D modeling.(3)A multi-view graph neural network(MV-GNN)is proposed to reduce the compression artifacts in multi-view compressed color images;a multi-view image fusion mechanism is designed,which make full use of contributions from neighboring viewpoints and meanwhile suppress the misleading information;a GNN-based fusion mechanism is designed,and a new model is explored using GNN's aggregation and update mechanism in the fusion of cross-view information.Experiments show that our algorithm can obtain an average gain of 1.672 d B and 0.0242 on the PSNR and SSIM metrics,respectively.For the subjective evaluations,blocking effect in the compressed images are clearly suppressed and the damaged object boundary are better recovered.The experimental results demonstrate that our MV-GNN outperforms the state-of-theart methods.