New View Synthesis Of Single View Transparent Object Based On Deep Learning

View synthesis is the generation of a scene image from a specified or unknown viewpoint given one or more images with known viewpoints.The complex optical path formed by the refractive reflection of transparent objects makes the synthesis of new views for transparent objects challenging.At present,there are four main research methods for new view synthesis: the first method is to estimate the 3D model of the scene through multiple views,thereby generating new views.This method provides more information about the occluded area,but it is limited by the input of multiple images.The second method is to fill the content of the occluded part with the contextual information of the image.This method takes full advantage of the semantic information of the image,but artifacts appear when generating new views of transparent objects.The third method is to estimate the depth map of scene to synthesise new views of the scene,but the depth estimation module has poor generalization ability.The fourth method is based on image rendering,which synthesise new views by estimating the normal or the refractive reflection stream of the scene.This method is suitable for complex scenes with transparent objects.This paper proposes a method for synthesizing new view of a transparent object from a new set of viewpoints based on a single RGB image of a transparent object and the corresponding segmented image—an Encoder-Decoder network for normal estimation and texture extraction,which can render a picture of a transparent object with a new perspective in a known and arbitrary environment map.The innovation of this paper is to consider the complex optical path changes of transparent objects with the rotation of the viewpoint,and learn the characteristics of optical transmission from the surface color to the normal and the change of perspective on transparent objects through the encoder-decoder network.A texture extraction subnetwork is proposed to alleviate the contour loss phenomenon during normal map generation.In addition,this paper creates the first multi-view dataset of transparent objects with complex backgrounds.In this paper,three evaluation methods are used to compare with the existing research methods.The experimental results show that our method performs better on view synthesis of transparent objects in complex scenes using only a single-view image and the corresponding segmented image.This method can more accurately capture the lighting changes and details of transparent objects in complex scenes at new perspectives.The loss function designed in this paper only penalizes the mean squared error between the true normal map and the prediction normal map which lead to a situation where the normal prediction result may be significantly distorted,resulting in an increase in rendering error.Due to GPU memory limitations,this model is currently not suitable for introducing more geometric constraints,such as smoothing errors on adjacent normal vectors or other surface-based curvature guidance.In the future,we will consider compressing the model and optimizing the production results.