Semantic-based Irregular Scene Perception And Reconstruction

3D semantic scenes are the current research hotspot of search and rescue robots.In dangerous environments,the search and rescue robot is used to detect irregular scenes and establish a semantic scene graph to guide the robot's automatic positioning and navigation.Make it possible to avoid dangerous areas and plan a reasonable rescue route.This thesis addresses the problems of large data volume,long processing time,and insufficient 3D model library in the 3D point cloud semantic recognition method.According to the corresponding relationship between pixel points and 3D point cloud,a method of transferring 2D semantic information to point cloud to establish 3D semantic scene is proposed.The main research contents of this paper are as follows:(1)Process the 3D point cloud data to create a 3D scene.Step 1: Calibrate Kinect V2 based on Zhang Zhengyou's calibration method to ensure the accuracy of point cloud data.Step 2: Perform preprocessing operations such as filtering and denoising and voxelization down-sampling on the collected point cloud to provide high-quality point cloud data for subsequent scene reconstruction.Step 3: Use RANSAC to obtain the reference plane of the scene,obtain the point cloud clusters of the scene through DBSCAN and record the centroid of each cluster.Convex hull algorithm is used to simplify point cloud clustering and reduce the amount of reconstructed data.Step 4:Compare the effects of two common reconstruction methods,the greedy projection triangulation method and the Poisson surface reconstruction method,and select the Poisson surface reconstruction algorithm.(2)Perform semantic segmentation on the two-dimensional image,transfer the semantic information to the three-dimensional scene,and build the three-dimensional semantic scene.Step 1: Establish a model of the FCN network,and use transfer learning to build the model.FCN-Res Net50-COCO is used as the initial data set,and the optimized model is trained,which is used to obtain the result of image semantic segmentation.Step2: Through the correspondence between the pixels on the 2D plane and the 3D point cloud,the semantic information is transferred from the pixels to the point cloud through a Bayesbased progressive method to achieve the establishment of a three-dimensional semantic scene.(3)Verify the method proposed in this thesis through experiments.Step 1: Compare the point cloud data before and after preprocessing,verifying that the preprocessing can effectively filter out noise points.Step 2: Perform Poisson surface reconstruction on the point cloud data of the two scenes,and record the time to verify the effect of the 3D scene reconstruction.Step 3: The experimental verification of the semantic segmentation method of deep learning can obtain the semantic prediction result with higher accuracy within the specified time.Step 4: Transfer the semantic information of pixels to the 3D point cloud to establish a 3D semantic scene.Different types of objects in the scene can be displayed in corresponding colors,and the reconstruction time and the accuracy of semantic information transmission meet the expected values.This scene has been used by search and rescue robots,which can significantly improve the efficiency of search and rescue.