3D Reconstruction And Rendering Based On Deep Learning

The main research in the field of computer vision and computer graphics revolves around how to make a computer has human's visual ability.It mainly includes two aspects: one is threedimensional(3D)rendering,that is,mapping the objectively existing three-dimensional world into two-dimensional(2D)images through the visual system,and the second is 3D reconstruction,that is,understanding and realizing three-dimensional scenes form the twodimensional projection image captured by eyes or cameras.At present,with the rapid development of computer technology and artificial intelligence methods,image-based 3D reconstruction and rendering technology is becoming more and more popular in navigation and positioning,medical imaging,resource detection,industrial design,automatic driving,digital entertainment and smart city applications.Therefore,it is very important to propose an efficient and accurate 3D reconstruction and rendering model.The classic 3D reconstruction methods use depth sensing devices such as Kinect and RealSense cameras to capture images with depth information from multiple views to restore the complete three-dimensional structure of the object.However,in practical applications,scanning all surfaces of the reconstructed object is not only time-consuming and laborious,but is not always feasible,which results in reconstructing complete 3D shape of the object with the occluded regions and large holes.At the same time,processing multi-view images requires more computing power,which is not ideal in many applications that require real-time performance.In addition,traditional computer graphics rendering pipelines use discrete operations such as rasterization and visibility computation,making it difficult to explicitly establish the relationship between rendering parameters and projections,making reverse rendering a significantly challengeable problem.In view of the above problems,this thesis uses a deep learning network to design a 3D reconstruction and rendering network based on single-view images,which can realize 3D reconstruction and 3D rendering at the same time.The network proposed in this paper contains two parts: 3D reconstruction network and 3D rendering network.The reconstruction network consists of a convolutional neural network(Encoder)that extracts image features(including structural features and texture features,etc.)and two convolutional neural networks that are used to recover the object's shape-decoder and texture-decoder,respectively.In addition,the rendering network consists of three convolutional neural networks that implement the functions of 3D-downsampler,Projection-unit,and 2D-generator.First,the shape and texture features of the objects contained in a single image are extracted into a representation vector by reconstructing the Encoder of the network.Secondly,they are input into Shape-decoder and Texture-decoder respectively to obtain a Shape Voxel and a Texture Model of the object.Next,the two models are concatenated and input into the rendering network.The 3D-downsampler removes the redundant information in the voxel and texture space.After that,the 3D information is mapped to the two-dimensional space by the Projection-unit,and up-sampling is performed by the 2D-generator to recover the image details,thereby obtaining a high-resolution rendered image.In this paper,the data-driven method,using the deep learning method and synchronously training the convolutional neural network,realizes the operation of reconstructing the threedimensional structure based on single images and rendering 3D voxels.The trained network can be modularized.The reconstructing network can be used alone to reconstruct the complete three-dimensional structure through a single-view image of a project,and the three-dimensional structure can be also used independently to render 3D Voxels with different texture information,thereby achieving the desired results.Therefore,this thesis has contributed to both the 3D reconstruction and the 3D rendering.The network proposed in this thesis explicitly wide and enormous image data samples and the strong representation ability of deep learning,which provides a new solution and method for future research in related work.