Image/Point Cloud Registration And3D Reconstruction Based On Semantic Information

With the rapid development of information technology,the demand for three-dimensional models in various industries such as movies and games is increasing day by day,so three-dimensional reconstruction has become more and more important.The key to three-dimensional reconstruction is the registration of images or point clouds,that is,the search for overlaps in two frames of images or point clouds.Current methods for image or point cloud registration are mainly based on low-level visual features,using the distance between two descriptors as a metric to match two images or point clouds.However,for scenes with little texture,it is difficult to extract feature points.And in scenes with many repetitive structures,feature point matching based on local information is incapable.In view of the shortcomings of low-level visual features,and considering that any scene has its semantic information,which is richer than low-level visual features,this paper introduces high-level semantic information,in order to solve the problem of image/point cloud registration and three-dimensional reconstruction in scenes with little texture or many repetitive structures,which is more versatile and robust.Based on this,the main research work of this paper are as follows.(1)Based on the repetitive structures in images,image matching is converted into grid matching,which is further simplified into the problem of determining right shifts in rows and columns.Obtaining the correct repetitive structure grid matching through shifts in rows and columns,point and line segment matching results can be gotten based on the vertices and edges of the grids,thereby achieving hierarchical,coarse-to-fine image registration.Compared with traditional feature point extracting and matching,our method is based on global information of repetitive structures,realizing automatic detecting and matching of repetitive structures in images,avoiding dependence on local features.The experimental results show that our method can achieve accurate image registration for images in scenes with little texture or many repetitive structures that are difficult to match with feature point based methods.(2)Based on semantic information of a point cloud,each object is abstracted as a node,and the adjacent relationship between two objects is abstracted as an edge,then a data structure of graph is constructed and defined as semantic connection graph.Based on sub-graph matching between semantic connection graphs,object correspondences between two point clouds can be obtained,which can be combined with information such as the contours of corresponding objects to get enough and more accurate corresponding points,realizing hierarchical coarse registration of two point clouds.Thereafter,through the iterative closest point algorithm,fine registration of the point clouds is achieved,and the coarse-to-fine registration of the point clouds is completed.Our method for point cloud registration based on semantic information makes use of subgraph matching to solve the problem of coarse registration of point clouds,avoiding the extracting and matching of feature points.Our method is faster,more accurate and more efficient than traditional methods.(3)For scenes with little texture and many line segments,two-tuples of adjacent line segments and their descriptors are introduced.Through the matching of coplanar line segments and their intersection points,image registration with more versatility is achieved.In scenes containing repetitive structures,the method for image registration proposed here can be used as a supplement to the algorithm of image matching based on repetitive structures.Obtaining accurate point and line segment matching results,we estimate the relative poses based on the corresponding points.Then we calculate each ratio of two translation vector lengths based on the coplanar constraint among three adjacent images to solve the absolute pose of an image.And based on this,high-precision three-dimensional reconstruction is realized.Our algorithm of three-dimensional reconstruction based on point and line segment matching gets rid of the dependence on feature point extracting and matching.At the same time,since the line segments in real space cannot exist independently from planes,although our method is based on coplanar line segments,actually there is hardly any requirement for the positional relationship between line segments,and the scenes only need to contain enough line segments.Experiments show that our algorithm of image registration based on coplanar line segments and their intersections can not only match images with large changes of viewpoints,which traditional feature point based methods usually mismatch,but also match multi-modal images that feature point based methods can not match at all.Our algorithm of three-dimensional reconstruction based on point and line segment matching not only achieves or even exceeds the current best level of precision on existing data sets,but also reconstructs high-precision point clouds and line clouds,which refer to three-dimensional line segment models,based on real images that are difficult to deal with by other reconstruction methods.(4)Considering that scenes with many line segments often contain lots of planes,we further propose an algorithm of plane extraction based on three-dimensional line clouds.By scaling and translating,we map each three-dimensional line segment to a point on the gaussian reference sphere with a radius of 1.A set of coplanar threedimensional line segments in real space,correspond to several points on the gaussian reference sphere that should lie on the same plane passing through the center of the sphere.In particular,when the coplanar line segments are parallel,the points on the gaussian reference sphere will overlap each other and gather at one point.The problem of plane extraction in real space is transformed into the extraction of planes passing through the center of the gaussian reference sphere,and the extraction of concentrated points on the gaussian reference sphere.Inspired by Hough transform,through spherical uniform sampling and voting,we fit planes iteratively,realizing plane extraction based on three-dimensional line cloud.Our algorithm takes a three-dimensional line cloud as input and simplifies the problem of plane extraction through gaussian reference sphere mapping.Compared with the traditional algorithms of plane extraction based on point clouds,our method is faster and more efficient.This paper proposes systematic algorithms of image/point cloud registration and three-dimensional reconstruction based on semantic information for scenes with little texture or many repetitive structures that are difficult to deal with by traditional methods.