3D Reconstruction And Semantic Segmentation Of Indoor Scenes Based On Depth Image

With the improvement of science and technology,more and more intelligent mobile robots appear in our daily life.Its application scenario is mainly in the unstructured road with specific indoor environment such as workshop,warehouse and building.Whether the robot can complete the work efficiently mainly depends on the accurate mobile positioning and the environment interaction.However,indoor environment is a closed three-dimensional space and cannot rely on GPS to complete all positioning,so it is necessary to consider the reconstruction of the actual three-dimensional environment structure,so that the robot can better complete the perception of the environment.Considering the overall cost,monocular color camera and depth camera are mainly used as the main sensing equipment.Therefore,this thesis studies 3d reconstruction of indoor scenes based on depth images and semantic segmentation of objects in the scene.The main research contents are as follows:1)Propose a deep image repair method based on dictionary learningThe image preprocessing involved in 3D reconstruction is studied to accurately construct the surrounding environment model to achieve better positioning.The deep learning network is used to detect the cavity and the cavity of the deep image is filled based on dictionary learning,which makes the camera pose calculation more accurate.A simple 3D reconstruction method using Kinect sensor is proposed.In the process of 3D reconstruction,it is inevitable that there will be cavities in space shielding and surface morphology of various objects.The first to use YOLO in front on the depth of acquisition of the single frame image empty area detection,using dictionary learning algorithm to fill empty to repair after,on this basis,using the iterative closest point(ICP)algorithm to further improve the camera pose estimation precision,finally combining truncation symbol distance function(TSDF)algorithm implements a three-dimensional reconstruction method.2)Propose a semantic segmentation method for indoor scenes based on fusion depth characteristicsAiming at the semantic segmentation of objects in the scene,this thesis designs a convolutional neural network(CNNs)model that can fuse multiple input features to provide semantic information for algorithm decision-making by perceiving and understanding unknown environment.The accuracy of comparing similar networks in NYU V2 data set,Scan Net20 data set and self-collected data set is improved to some extent.A semantic segmentation model of CNNs based on color image,depth image and HHA(horizontal parallax,height above the ground and local surface normal vector)is designed,so that color image and depth information can be used to jointly predict semantic information.The proposed model reached an average of 67.3% in THE NYU V2 data set.MIOU reached 80.1% on the Scan Net20 dataset.Compared with similar methods,the proposed method has a better accuracy in scene semantic segmentation.