Research On Scene Understanding Technology Based On Deep Learning

3D scene understanding is a very popular research in the field of Computer Vision and Computer Graphics.It has been widely used in Augmented Reality,game production and other industries.The surface normal estimation of the scene plays an important role in 3D scene analysis,2.5D layout description of the scene,and illumination information extraction.However,the existing surface normal estimation methods usually cannot obtain the overall distribution of the data,and the estimated surface normal is often blurred and the accuracy is low.In order to solve these problems,this thesis designs a normal estimation model based on Generative Adversarial Networks.Moreover,combined with the spherical harmonic function,the extraction of illumination from a single RGB image and the combination of virtual and real objects are completed.The main work of the thesis is as follows:(1)A normal estimation model based on Generative Adversarial Networks is designed to realize the accurate estimation of the surface normal in 3D scene understanding.After using a baseline model to explore the normal estimation task,aiming at the difficulty in extracting layout features of the scene,such as the position of the ceiling,a global feature network is designed,and the self-attention mechanism is added to further seek the global dependence of the feature,then the overall layout feature information of the scene is extracted.In addition,in order to solve the inverse problem in the normal prediction of the vertical plane,a prior conditional network is designed to provide the layout standard of indoor normal,which accelerates the convergence of the network while solving the problem.In the loss function part,the basic model calculates the loss of traditional Generative Adversarial Networks and the Manhattan distance between real images and the generated images.Then we add an angle difference loss at the pixel level,so the estimation result is improved.The designed algorithm is applied to the standard data set NYU Depth V2.The evaluation was carried out to verify the rationality and effectiveness of the algorithm design.(2)Aiming at the problem that depth information often has a lot of noise in traditional illumination extraction methods,based on the above work,this thesis can extracte the illumination from a single RGB image directly,using the accurate surface normal obtained from the image combined with the spherical harmonic function.A virtual object is then created with the acquired illumination information,and it is integrated into the original 2D RGB image.So the combination of virtual objects and real objects with illumination consistency is completed.It further proves that the algorithms designed in this thesis used in surface normal estimation and illumination information extraction task are effective.