Autonomous 3D Reconstruction Of Unknown Indoor Scenes

Since George Devol invented the first programmable robot in 1954,robots have accompanied humans for more than half a century and become increasingly indispensable in production and life.In order to endow the robots with the ability to parse surroundings and operate objects like humans,one of the pivotal steps is to make them capable of accurately sensing and reconstructing the 3D objects and environments around them.While there has been a rise in robust online 3D reconstruction algorithms,most existing algorithms assume a human scanner with a hand-held camera.How to automatically conduct scanning and reconstruction with robots is still an unsolved and challenging research topic.To carry out an autonomous 3D scanning and online reconstruction system for unknown scenes,one has to strike the balance between global exploration of the entire scene and local scanning of objects within it.To solve this problem,we first propose a novel autoscanning approach using object-aware guidance,which can accomplish exploration,reconstruction,and semantic understanding of an unknown scene within one navigation pass.Our approach interleaves object analysis to identify the next-best-object(NBO)for global exploration,with object-aware information gain analysis to plan the next-best-view(NBV)for local scanning.Besides,this method couples object segmentation with object recognition via a multi-class graph cuts minimization with an objectness metric to achieve object recognition during the scanning process.The thorough experiments and comparisons have shown that the proposed autonomous 3D reconstruction system can reconstruct and understand the unknown indoor scenes within a single navigation pass with high coverage and high quality.Existing autonomous 3D reconstruction approaches rely on hand-crafted feature extraction and scanning strategy in a detached manner.Besides,these methods fail to employ high-level semantics other than information from a single object.These all hinder the efficiency of the planning.We further propose a deep reinforcement learningbased(DRL-based)hierarchical global-to-local scanning strategy.To overcome the difficulty in training an end-to-end scanning policy purely based on 3D representation,the proposed method exploits an innovative mixed 2D-3D representation.A reconstructionaware global 2D map encodes object scanning quality and semantic information of the current perceived scene structures.While a coarse-to-fine local 3D map encodes the geometric information of multiple objects and the region encompassing them on a small scale.Based on this mixed representation,the proposed method conducts the scanning by alternatively utilizing a global scanning policy to specify a region-of-interest(ROI)and a local scanning policy to generate the NBV.Through comprehensive experiments,we show that the proposed DRL-based scanning strategy can explore the unknown environments more efficiently and obtain high-quality object reconstruction at the same time.Due to the huge search space of the scanning strategy,it takes a long time to train the DRL-based autonomous 3D reconstruction method.We,therefore,introduce two targeted approaches to improve the training process.Firstly,we design two auxiliary learning tasks,which are pertinent to the prediction of path complexity and the recollection of scanning progress,to provide additional supervisory signals for the training of the global scanning policy and promote understanding of spatial and temporal relations.Secondly,a path segmentation-based experience enhancement method is also introduced.This method is designed to help the agent extract correct actions from failures and accelerate the exploration of high return policies.Through various comparative studies,we validate that these two improvements can efficaciously improve the training efficiency and final performance of the global scanning policy.This paper showcases the viability of jointly solving the scene exploration,semantic understanding,and scene reconstruction,as well as the superiority of applying deep reinforcement learning to the autonomous 3D reconstruction.By making full use of the object-level and high-level semantic clues,our method manages to scan the scenes efficiently and obtains high-quality reconstruction results.Our systems are tested completely in the standard ROS and Habitat environments along with a real robot and compared sufficiently with various open-source baselines,which demonstrates the feasibility and effectiveness of our method.