Research On Construction Of Visual Semantic Map Based On 3D Point Cloud Deep Learning

Visual SLAM is the key technology for intelligent robots and driverless cars to realize fully autonomous navigation in unknown environments.The traditional visual SLAM focuses on geometric location and mapping,but has no semantic information of the environment.Therefore,it is a hot topic to construct semantic SLAM with semantic information map based on it.In the existing semantic SLAM scheme,the semantic information is mainly acquired based on the target detection or semantic segmentation of two-dimensional images,and then the acquired semantic information is mapped to the 3D point cloud segmented by clustering,so as to indirectly obtain the semantic information of 3D point cloud corresponding to the corresponding frame.The method of fetching the target semantic information is not based on the direct feature extraction of the original 3D point cloud,and there are some problems such as indirect processing method,poor semantic effect,insufficient depth information,and fuzzy 3D edge segmentation.In addition,the traditional visual SLAM and the existing semantic SLAM schemes are mainly limited to the static environment and cannot be applied to the complex dynamic scenarios.Therefore,aiming at the shortcomings of the above research scheme,this thesis studies the construction method of visual semantic map based on 3D point cloud deep learning.Using RGB-D sensor information,the dynamic object detection is firstly added on the basic framework of ORB-SLAM2,Then the 3D point cloud semantic segmentation method is used to obtain scene semantic information,and finally the 3D semantic map after dynamic object detection and elimination is constructed by integrating the semantic information.The constructed semantic map can be used for the location and navigation of indoor service robots and the semantic understanding of scenes,etc.The main work and achievements of the thesis are as follows:(1)This thesis analyzes the basic theory and algorithm framework of visual SLAM front and back end,briefly describes the semantic acquisition idea and map construction method of semantic SLAM,and designs the overall research scheme of semantic map construction according to the different scene adaptability and semantic realization schemes.(2)In order to study the visual SLAM based on RGB-D sensor,a dynamic object detection module is added to ORB-SLAM2 visual SLAM framework,and a visual semantic map construction method based on 3D point cloud deep learning is proposed.First,through collecting Kinect camera device colour image and depth image.Then,the improved ORB feature extraction algorithm is utilized to extract the reference frame and current frame image of the ORB feature information,and adjacent frames features are matched to initialize the position and relocation.Then,the possible dynamic objects are pre-screened through 3D point cloud semantic information.Finally,the inter-frame camera motion consistency algorithm is used to exclude camera motion,then the dynamic threshold is set through the optical flow method to check whether the feature points of the possible moving objects are moving or not.Due to the limitation of the effect of optical flow method,this thesis designs and implements the Multi-angle geometric dynamic points detection algorithm based on 3D vision,whose dynamic feature point detection effect is better than that of optical flow method,and improves the dynamic scene adaptability of the ORB-SLAM2.(3)Aiming at the limitation of existing SLAM schemes in 3D semantic information acquisition methods,the semantic segmentation method for feature extraction of 3D point cloud is studied,and a semantic information acquisition algorithm based on 3D point cloud deep learning is proposed.First,on the basis of point cloud deep learning network Point Net,the feature extraction module of MLP structure in the network is replaced by the feature extraction method of dynamic graph convolution,which strengthens the network's ability to learn the feature information of adjacent points.Then,a spatial pyramid pooling structure based on point cloud is designed,which enhances the network's ability to learn and extract fine-grained features.Finally,the semantic information obtained is used for the subsequent semantic library construction and semantic map fusion in the semantic SLAM system.(4)Research on update and merge of map and semantic information after removing dynamic objects,including dynamic semantic map construction algorithm process and system overall design,the construction of semantic information database,through semantic fusion algorithm for dynamic semantics octree map building and share update,local map and global map of bundle adjustment optimization.In addition,the construction of octree semantic map is studied,construct a semantic map suitable for indoor visual location of service robots,navigation and scene semantic understanding.