Research On 360° Virtual View Video Generation Technology

Virtual viewpoint video synthesis technology is a technology that synthesizes new viewpoint video of the 3D scene video based on reference videos captured synchronously by a series of cameras.It is a hot issue in the field of virtual reality and multimedia technology.Virtual viewpoint video synthesis technology based on 360 ° panorama combines the immersion brought by panorama technology and the interaction brought by virtual viewpoint synthesis technology,so that users can better observe the situation of the scene and bring immersion experience.Because cylindrical panorama has many advantages,such as wide application range,easy access and easy processing,the author aims to study 360° virtual viewpoint video synthesis technology in the form of cylindrical panorama.In this paper,the imaging principle of 360 ° virtual viewpoint video,virtual viewpoint synthesis based on optical flow,subregional virtual viewpoint optimization,texture optimization based on line matching and flow optimization based on Mask R-CNN are studied.The main innovations are as follows:1.A subregional 360°virtual viewpoint generation algorithm based on optical flow is proposed.Without any camera parameters and other geometric information,it can be accomplished only according to reference viewpoint videos,which greatly reduces the cost of solving the problem.The combination of optical flow pyramid and block matching achieves a good effect under large parallax.In view of the characteristics of cylindrical panorama,a new view synthesis method based on sub-region is proposed,which solves the problem that the deformation of cylindrical panorama is too large to match.2.Aiming at the problem that the effect of linear texture area is not good,an optical flow optimization method based on line matching is proposed,which solves the linear characteristics deformation caused by full search block matching.Firstly,feature point matching is used to cut the reference viewpoint into regions with the same image content.Then,line matching technology is applied to the corresponding regions,and the optical flow of the regions is corrected according to the results of line matching.The method corrects the optical flow in the straight line and the area around the straight line,and effectively improves the effect of virtual viewpoint generation in the straight line area.3.To solve the problem of object shape distortion or object edge and background mixing,a segmentation method based on Mask R-CNN network trained by COCO dataset is used to segment the main objects in the image,and optical flow optimization is added to the optical flow calculation process to further improve the quality of virtual viewpoint.Experiments show that the 360°virtual viewpoint video generation method based on optical flow proposed can generate better virtual viewpoint video in most indoor and outdoor scenes.