2D-2D/3D Visual Feature Matching And Its Applications

Currently,the fusion analysis of 2D imagery and 3D point clouds has become a research hotspot in the fields of computer vision and robotics,while the 2D-2D/3D registration is the basis for visual data fusion.The general idea of 2D-2D/3D visual registration is to establish the geometric or photo-metric correspondences between different visual data in the same scene,which can be used for visual data transformation and fusion.It is widely used in the field of remote sensing observation and robotic navigation,etc.Focusing on the 2D-2D/3D feature matching,this dissertation studies the 4 topics:image-level feature description for 2D-2D global registration,local 2D feature extraction and matching for 2D-2D local registration,2D-3D line feature extraction and matching for 2D-3D global registration,and the online line feature matching for 2D-3D local registration.Experiments on a variety of 2D images and 3D point clouds demonstrate the reliability of the proposed methods,thus a systematic framework is established for 2D-2D/3D registration.The scientific contributions are listed as follows:(1)On the problem of the 2D-2D global registration,an image-level descriptor(called CTS)is proposed to effectively encode color,texture and structure information for improving its accuracy and discrimination.By utilizing the binary partition tree structure of images,the descriptor is able to extract and fuse multi-scale and multi-level features.The performance of the proposed descriptor is tested on the tasks of semantic object matching and scene classification of high-resolution remote sensing images,experimental results demonstrate it has high description accuracy and discriminability.(2)In terms of 2D-2D local registration,an framework of feature extraction and matching is proposed for 2D-2D local registration to improve the robustness and efficiency.To improving the registration accuracy of SAR images,a SAR-FAST corner detection algorithm and a DSP-LATCH description method are proposed.For feature matching,the Local Sensitive Hash search is used to replace the kd-tree search strategy,which makes the nearest neighbor matching more efficient.The matching efficiency is verified in the task of UAV image stitching,and the matching performance based on SAR-FAST and DSP-LATCH is evaluated in the experiments of bi-temporal SAR image registration.(3)A 2D-3D global registration method is proposed for the alignment of images and point clouds in structured environments,which can simultaneously estimate 2D-3D line correspondences and camera poses.To address the 2D-3D appearance differences and modality gaps,2D and 3D line features are utilized instead of point features to ensure the feature repeatability.Specifically,the geometric constraint between vanishing points and 3D parallel lines is used to compute all feasible camera rotations.Then,a hypothesis testing strategy is utilized to estimate the 2D-3D line correspondences and the translation vector.By checking the consistency with computed correspondences,the best rotation matrix can be found.Finally,the camera pose is further optimized by minimizing the reprojection error of all 2D-3D line correspondences.The experimental results on both synthetic and real datasets demonstrate the effectiveness of the proposed method.(4)For the task of real-time camera localization in 3D point cloud maps,a local 2D-3D line matching method is proposed to directly estimate the 2D-3D line correspondences and camera poses.To handle the 2D-3D appearance differences and modality gaps,geometric 3D lines are extracted offline from 3D maps while 2D lines are extracted online from video sequences.With the pose prediction from visual odometry,local visible 3D lines in camera field-of-view can be efficiently extracted and matched with 2D line features.Then the camera poses and 2D-3D correspondences are iteratively optimized by minimizing the projection error of correspondences and rejecting outliers in the sliding window.Experimental results on the Euroc Mav dataset and the Realsense-collected dataset demonstrate that the proposed method can efficiently obtain stable 2D-3D line correspondences and estimate camera poses.