Visual Semantic SLAM Based On Deep Learning

Classic Simultaneous Localization and Mapping(SLAM)has now developed into a mature technology.In order to perform high-level tasks more smartly,robots should perceive semantic information of the surrounding environment.Semantic SLAM is a new topic in the field of SLAM.The rapid development of deep learning has provided good conditions for semantic SLAM.SLAM and semantic perception methods can promote each other.SLAM can provide spatial geometry information while semantic perception methods enable high-level understanding of the map.Several problems have been studied in the thesis,they are: applying deep neural networks into visual odometry,fusing semantic information in the level of object into SLAM and building semantic SLAM systems.Main works of this thesis are as follows:1.A visual odometry based on deep local feature DELF is designed and implemented.Most classic SLAM systems are based on artificial features such as SIFT and ORB.Artificial features don't work well in complex environments.It has been validated by experiments that the DELF feature based on deep convolutional neural network can help to improve the performance of the visual odometry.2.A novel semantic mapping method based on RGB-D SLAM and frustum semantic segmentation is proposed.Most semantic mapping methods build dense maps and extract 2D semantic information,which are followed by mapping and refinement steps.In this thesis,a sparse map is built firstly,and a 2D object detection method is performed on key frames.Next,depth and pose information are used to get point clouds in the frustum.Semantic segmentation is directly performed on 3D point clouds.A Bayesian Update scheme is designed to fuse information from multiple frames.It has been validated that this method enables efficient semantic mapping and that SLAM promotes semantic perception.3.A semantic DELF-SLAM system based on semantic data association is proposed.DELF feature is used for feature matching,re-localization and loop closure detection.The objects of interest play their role in the process of selecting key frames and nonlinear optimization.Semantic information acts as a factor in nonlinear optimization.Sequence matching method is introduced to validate a loop closure.It has been validated that each module helps to improve the performance and that our method outperforms many classic methods.