Research On Real Time Semantic Mapping Based On UAV Visual Localization

UAVs(Unmanned Aerial Vehicles)have played a significant role in serving our society such as inspection and rescue due to their flexibility recently.As the complexity of tasks increases,the requirements for UAV intelligence are becoming higher.Making the UAVs capable of scene understanding is a fundamental requirement for them to perform full autonomous tasks.With the rapid advances in visual SLAM(Simultaneous Localization and Mapping)and semantic segmentation,the semantic mapping generated by the combination of these two works makes UAVs an effective approach for scene understanding.Building 3D maps with semantic information in real time enable UAVs to effectively make reasonable action plans to avoid obstacles during flight,and can further identify objects and scenes to complete advanced tasks.However,integrating simultaneous real-time localization,3D reconstruction,and semantic segmentation is a huge challenge for systems with limited power like UAVs.To address the above problems,this article conducts research on the integration of semantic segmentation network and 3D mapping module on a visual SLAM framework to achieve real-time semantic mapping based on systems with limited computational resources such as UAVs.The main work of this paper is as follows.1.A real time semantic mapping system is constructed,which uses visual SLAM as the base framework,a deep learning-based semantic segmentation network,and a real-time dense 3D mapping module are embedded in the back-end part of the base framework.2.In order to reduce the computation process of ORB feature point extracting and matching in the front-end of the visual SLAM framework,a pose solving approach based on the direct method is introduced to accelerate the tracking speed of the vision odometer while guaranteeing the localization accuracy.3.A set of indoor image data is captured by using a UAV and a labeled dataset is created.Models of lightweight semantic segmentation networks such as Seg Net,ICNet,Deep Lab V3+,and Bi Se Net V2 are trained and tested by using this dataset to select the most suitable network in terms of accuracy and speed for embedding into the semantic mapping system.4.A dense mapping module is designed in the back-end part to complete the 3D map building task.The module generates a local map from color and depth maps,then colors the map with semantic information and finally stitches it with camera poses to produce a global 3D map.The 3D map uses an octree representation that is more efficient than point cloud maps for storage,can be used for navigation and can be updated at any time.5.A quadcopter UAV platform is built,this article designs the hardware structure of the UAV and configures the software part.The performance of the system is tested on this platform and the results show that the system has the ability to build large scale semantic maps in real time on platforms with limited computing resources such as UAVs.