Multimodal Remote Sensing Image Matching Based On Deep Features

In recent years,with the rapid development of remote sensing technology,the data volume of remote sensing information has increased exponentially.The emergence of multiple sensors and data types has also brought great opportunities and challenges to remote sensing data processing and applications.The differences in imaging mechanisms of different sensors enable multimodal remote sensing images to reflect different characteristics and information of ground objects.Therefore,fusion of multimodal remote sensing images and making full use of the complementary advantages of various types of remote sensing data will be beneficial to image interpretation and information detection.In multimodal remote sensing image fusion,image matching is a critical step.The existing methods for multimodal remote sensing image matching are mainly divided into two categories: area-based methods and feature-based methods.Among them,featurebased methods can be subdivided into hand-designed descriptor based methods and learningbased methods.In area-based method,a template window and a search window on the reference image and the target image are established at the same time,and the constructed similarity metric for nonlinear gray-scale differences is utilized to measure the statistical values in the window,then the matching point will be determined by the measurement.In the category of feature-based matching methods,methods based on hand-designed descriptors provide an abstract representation of image grayscale by constructing feature descriptors,which is robust to nonlinear grayscale differences between multimodal remote sensing images.While the learning-based methods extract deep semantic features that are robust to the nonlinear gray differences of multimodal remote sensing images by constructing and training a deep convolutional neural network.Then the similarity metric constructed manually or directly learned by the network is applied in the comparison between the deep image features.However,due to the difference of sensors and platforms,there are commonly significant nonlinear grayscale differences and geometric deformation among multimodal remote sensing images.Although many methods have been proposed in the field of multimodal image matching,it is still difficult to achieve reliable matching performance between images with significant nonlinear grayscale variation and geometric distortions.Therefore,in this thesis,a more in-depth study is performed on the multimodal remote sensing image matching method based on deep features.This thesis focuses on the feature matching of multimodal remote sensing images based on deep learning and conducts in-depth research on the matching difficulties caused by nonlinear grayscale differences and geometric deformation in feature matching of multimodal remote sensing images.The research content is mainly divided into the following three aspects:(1)Construction strategy of multimodal remote sensing image feature matching sample dataset considering sample distanceLearning-based methods require a large number of samples for model training,but in the field of multimodal remote sensing image matching,there are few public datasets that meet the requirements of this research.In addition,due to the similarity of neighboring features on the image,it is easy to cause mismatching by neighboring points around the correspondence.Therefore,a construction strategy of multimodal remote sensing image feature matching sample sets is proposed in this study by taking the sample distance into account to overcome the negative influence of neighboring points in image matching.(2)Deep feature extraction of multimodal remote sensing imagesThere are significant nonlinear grayscale differences between multimodal remote sensing images,which seriously affects the matching precision between images.It is observed in our study that the nonlinear grayscale differences between deep features of multimodal images extracted through convolutional neural networks are largely eliminated.Therefore,a convolutional neural network with dual branches and shared weights based on the Siamese architecture is applied to extract deep features that are robust to nonlinear grayscale differences.This network is constructed and trained for the subsequent feature matching.(3)Image matching framework based on deep featuresAlthough the nonlinear grayscale differences between the deep features extracted by the convolutional neural network can be largely eliminated,there are still geometric deformations between multimodal images.In addition,the optimal pair of deep features with similar grayscale distribution should be selected from the large amount of extracted feature images.In consideration of these problems,a robust deep feature based multimodal image matching framework is proposed.First,deep feature maps with similar grayscale distribution are selected using a Bo F-based image retrieval method.A geometric transformation is estimated based on the selected deep feature maps.And then,coarsely registration is performed on the target image to eliminate the geometric distortion.After that,local feature patches are extracted from the coarsely corrected image pairs and sent into the aforementioned matching network to find matches.Finally,these matches are computed back to the original images to get the final matching result.In summary,this research proposes a feature matching framework which is robust to both nonlinear grayscale differences and geometric deformations between multimodal images to improve the matching performance of multimodal remote sensing images.