Learning Based Novel View Synthesis On Monocular Images

As a popular research in the filed of computer vision,novel view synthesis has a wide range of applications,such as virtual reality,3D display,2D to 3D video transformations and so on,providing more vivid and acceptable displaying approaches for the traditional image and video media.Novel view synthesis is defined as rendering novel views for a specific scene,given some known views of the same scene.As a key technique,depth estimation provides the geometry of the scene for sampling and rendering to generate novel views.As a result,the quality of depth estimation affects the visual effect of novel view.The traditional multi-view-based depth estimation methods are mainly based on feature point matching,whose accuracy is limited in the lighting-unconsistency and textureless regions which usually appear in the complex scene of the real-world natural images.More over,the multi-view-based methods usually require multiple inputs which are not easy for collecting in practice.The existing learning-based depth estimation methods take advantages of abundant data for scene understanding and in some aspects make up the accuracy decrease in the tricky regions mentioned above and even monocular images can be used for depth estimation.However,these learning-based methods usually output depth results which are over-smooth,containing less details and the resolution of outputs is small,which should be up-sampled during the high resolution synthesizing process.Naive up-sampling process introduces edge aliasing.In this paper we systematically research and analyze the learning-based depth estimation and novel view synthesis.Considering the constraints in existing work,we propose an approach to improve the depth estimation results via detail enhancement and edge refinement during up-sampling.The main research contents and contributions of this paper are :(1)A multi-detail-scale input constructing method for learning.Inspiring by the image enhancement researches,we firstly compute multiple detail images and the enhanced images for the RGB training data and combining these images together as the network input for depth estimation.This process provides more details for the neural network and the network can adaptively choose detail features to enrich the depth detail.(2)A multi-level loss function for learning.Inspiring by the image style transfer researches,we design a multi-level loss function to constrain the outputs via multiple aspects including color,content and texture.(3)A learning-based depth upsampling method guided by the RGB image.Via encoding the RGB image and decode it together with the low resolution depth image,we can get the high resolution depth result.More over,we apply the edge detection method to detect the object boundaries and convert them into the weights for the loss function,making the network pay more attentions on the object boundaries and output finer results.In this paper,qualitative and quantitative analysis are used to verify the effectiveness of the proposed method in depth estimation and view synthesis optimization.