Selecting Initial Image Pairs For Incremental 3D Reconstruction Based On Multi-task Learning

The 3D modeling of real scene plays an essential role in urban construction,highprecision map and geographical survey.Therefore,the research on Image-baesd highprecision 3D modeling has gradually became the focus of researchers,and a great deal of methods have emerged.Incremental 3D reconstruction is widely used in the field of academic research and engineering due to its good robustness and high modeling accuracy.Among them,the initial image pair selection method in traditional incremental reconstruction consists of similar image search and relative position relationship calculation.They all rely on the efficiency and accuracy of feature point extraction such as SIFT and SURF.In addition,this process also requires feature point matching between pairwise images.This process not only has a large number of redundant pairings,leading to many unnecessary calculations,but also not robust enough in the face of poor lighting conditions and blurring.In view of this problem,deep learning method has been applied in many computer vision fields in recent years because of its advantages of small resource occupancy and fast computing speed.This paper also tries to use deep learning method to improve the problems existing in traditional initial pair selecting of incremental reconstruction.This paper introduces the basic theory of multi-task learning and the main algorithm of incremental 3D reconstruction,and analyzes the shortcomings of the traditional initial image selection algorithm based on feature matching,explaining the reason why it takes too long to calculate.Based on the analysis and research of the above problems,this paper mainly completes the following tasks:Firstly,the traditional incremental sparse reconstruction method is briefly reviewed,then we proposed the initial image pair scoring neural network based on the regression neural network.Through the traditional structure-from-motion pipeline we generate the data sets,and we verify the feasibility of using neural network for initial image selection by test experiment.Secondly,based on the theory of multi-task learning and deep neural network,this paper proposes a multi-task network framework that can simultaneously predict image similarity and image camera position and attitude.By learning image features and geometric information at the same time,the whole network model become more efficient and stable.This paper further studies the sharing mode of parameters among multi-task networks,then we proposes the initial pair selection network of various modes,including parallel,shallow cross connection and deep cross connection,and explains the most appropriate connection mode for the initial pair selection problem.Experiments show that the multi-task initial image pair selection network proposed in this paper,can efficiently obtain the initial image pair of incremental 3D reconstruction,greatly reducing the computation time.Meanwhile,compared with the traditional initial image selection algorithm,the accuracy of sparse reconstruction is also slightly improved.Thirdly,in order to avoid the problem that the reconstruction scene is incomplete due to the selected initial image pair is located in a sparse area a,the intermediate results of the multi-task learning method are used to calculate the scene connection graph,and an initial image pair selecting strategy based on the scene connection graph is proposed.The experiments show that the strategy can well guarantee the robustness of 3D reconstruction in special scenes.