Illumination-aware New Viewpoints Synthesis Based On Deep Learning

New viewpoint synthesis,also known as novel viewpoint drawing,is to synthesis image at the position of other virtual viewpoint or camera in scene using images taken by camera at one or more different positions(viewpoint).Virtual viewpoint rendering methods are mainly divided into two kind: based rendering(MBR)and image based rendering(IBR).Atually,IBR uses limited but efficient viewpoint image information,while MBR utilizes a large amount of viewpoint image information for complex geometric modeling and calculation and achieves a true three-dimensional immersion.New viewpoint rendering technique can be used for applications such as virtual reality,image stabilization or monocular 3D movie.MBR can synthesis accurate viewpoint in any direction while it needs a large number of complex input viewpoint information for long-term calculation,which limits its application.IBR can synthesis limited and efficient viewpoints quickly but depends on the finite calibrated viewpoint information,which makes IBR hard to achieve viewpoint switching in any direction,also the color inaccuracies of specular component in the synthesized images in the traditional IBR method are obvious.This paper proposes a new free viewpoint synthesising technique that avoids the modeling and calculation of a large amount of viewpoint information in the MBR,but selects the rapid synthesising method IBR which is based on the viewpoint mapping relationship,and constructs a geometric model of the scene to synthesi free viewpoints in any direction.At the same time,there is an inherent premise in the traditional IBR synthesis technique,this problem has been effectively solved in the MBR by considering light in the model,the same object which is projected to the virtual viewpoint in reference viewpoint and virtual viewpoint share the same color,which results in the higher the specular optical coefficient of the material contained in the scene,the more significant the unrealistic result of the synthesis is.Thus we revise it.The idea of traditional IBR only considers ambient light component and diffuse reflection component which are perspective-independent components in the illumination model,but ignores the specular component associated with the viewpoint,so that when the viewpoint is swtching especially to the viewpoint in any direction we proposed the specular component of the viewpoint image is s?tatic?nd thus seriously affects the real sense of three-dimensional.Our approach uses deep neural networks to optimize the display of the specular component of color information under the final virtual viewpoints,due to the IBR's own property of limited input information it is difficult to modeling this problem in IBR which is relatively complex problem rather than modeling in MBR,so we use neural networks for its powerful learning ability of nonlinear processes and build a suitable neural network structure to solve this problem.1.Without using depth map,analyzes and summarizes the process of IBR synthesis technique,and combine depth estimation and viewpoint synthesis in end-to-end depth con-volution neural network.2.Revise inaccuracies in traditional IBR synthesis results,especially the reflection component.3.Synthesis free-viewpoint at any direction based on geometric model and use CUDA to accelerate the entire synthesising and repairing process to achieve real-time video viewing with viewpoint transforming.In the end,this paper proves through experiments and tests the accuracy of our understanding of IBR,and we synthesis virtual viewpoint in End-To-End CNN while the performance of synthesised specular component of color under virtual viewpoints is increased.Finally we achieve the technology of the fast and efficient synthesising of free-viewpoint in any direction.