Study On Virtual View Synthesis Based On Depth Information

Free viewpoint video(FVV)is capable to provide viewers with arbitrary viewing angles and viewing locations,which brings a desirable human-machine interaction and watching immersion,therefore FVV is considered to be one of the main future direction of digital TV.FVV can be widely used in entertainment and education industries such as sports events,cultural performances,interactive courses,etc.It can also be applied to social work such as clinical surgery,military strategy research and national defense technology.The development of FVV can improve the production efficiency of society and change the life of human beings.Its development is of great significance.However,As the number of viewable viewpoints increases,the video data also increases exponentially,which puts tremendous pressure on the collection,storage and transmission of video.Virtual view synthesis technology is to solve this problem.It uses the captured scene information to render the image acquired at the virtual viewpoint,which can effectively reduce the number of viewpoints that need to be acquired,making FVV easy to popularize.Obviously,virtual view synthesis is a key technology for free viewpoint video.The main difficulties encountered in view synthesis process are: the collection of depth information will increase the cost of FVV production;the appllied depth information is very coarse,leading to unsatisfied image quality;large empty regions are often introduced by the disocclusion phenomenon and so on.This paper puts forward several key technologies to deal with these problems,designs several rendering frameworks,and verifies the feasibility and superiority of the proposed method through experiments.The main contents and research achievements can be summarized as follows:Firstly,an adaptive depth estimation algorithm based on local stereo matching is proposed.In order to overcome the problem of high mismatch rate of local stereo matching algorithm,an algorithm with the functionality of adjusting the matching window size according to the cost aggregation situation is designed.The experimental results show that the proposed method is close to the traditional local stereo matching algorithm in operation speed,and the mismatch rate is reduced by half.While the matching accuracy is close to the pixel-based multi-measure fusion method,the running time is only about 15% of the latter.The performance of the adaptive depth estimation algorithm in terms of computational speed and accuracy meets the needs of practical applications,and is of great significance for the promotion of FVV.Secondly,this thesis analyzes the influence of depth information on the virtual image quality,study and verify the performance of the adaptive trilate filter and the feature-based depth refinement method,then proposes a depth refinement based on left and right consistency verification.This method divides the depth pixels into three categories: high reliability,low reliability,and undetectable.The spatial correlation of the depth map is used to refine the unreliable pixel depth values to obtain a corrected depth map.Experimental data shows that the proposed method reduces the number of holes by 2% to 6% compared to the previously proposed methods.The PSNR increased by 0.2 dB to 0.5 dB.Thirdly,due to the fact that the virtual viewpoint and the reference viewpoint are different in angle and position when shooting the scene,large empty areas are introduced during the rendering process.After studying the methods of using spatial information to fix this problem,a global background modeling method based on depth information is proposed.This method utilizes depth information to avoid detecting recurring foreground objects as background regions,thereby obtaining a more accurate global background image.PSNR shows that the temporal method gains 2.2 dB and 0.7 dB increase above the traditional inpainting technique and the conventional GMM method respectively.At last,a depth map processing method based on updating the depth map information at the virtual viewpoint is proposed to deal with noises.The method performs denoising processing only on the detected pixels,this can effectively eliminate various artifacts and noises,and preserves the boundary information and texture details in the entire image.In addition,a variety of advantageous rendering frameworks have been designed.The low-latency synthesis framework are balanced in rendering quality and computation speed,the layered mapping framework only needs reference information from one viewpoint,and better objective evaluation is achieved by the reliability-based rendering technique.In this paper,the virtual view synthesis technologies for free viewpoint video are studied,explored and verified.Several techniques based on spatial and temporal correlations are proposed to solve the problems in viewpoint rendering process,which has guiding significance for the development of video technology.