Robust Registration Of Multimodal Images Based On Local Patch Matching

Image registration is widely used in many computer vision applications,such as image stitching and image fusion.Its purpose is to convert two images into the same coordinate system.In addition to the traditional challenges(illumination changes and geometric transformation),registration of multi-modal images faces significant appearance differences in pixel features caused by different imaging mechanisms.Thus,registration of multi-modal images is more challenging than the ordinary registration.This paper proposes three registration methods based on deep learning to solve the problem of multi-modal image registration.(1)This paper proposes a progressive comparison of spatially connected feature metric learning with a feature discrimination constrain(SCFDM)to study robust features from shared space.The progressive comparison of spatially connected feature network keeps the property of each domain in its corresponding low-level feature space,and integrates the cross domain features in a high-level feature space to obtain their semantic feature.Feature discrimination network constrains the feature network to extract shared features between two patches.The metric network analyses the extracted feature to predict whether the patch-pair is matched or not.Extensive experiments show that SCFDM can not only achieve a notable matching performance,but also can be successfully applied to registration of multi-modal images.(2)In view of the prominent generalization ability of L2-Net,this paper proposes an improved version for registration of multi-modal images.A symmetric loss is introduced based on L2-Net for studying shared features from two patches,which improvesthe robustness of the model,and increases the training efficiency as well.In addition,an adaptive crossentropy loss function is proposed to replace the log-likelihood in L2-Net for better searching for the optimal solutions.Experiments have proved that our method is far superior to L2-Net in both local patch matching and multi-modal registration.(3)In order to further reduce the registration error,a novel loss,called Soft Reg loss,is proposed.It can adjust penalties according to different proportion of texture overlap between patches.Moreover,a balanced hard mining strategy is designed based on symmetric loss which makes the model focus more on hard negatives and positives.Extensive experiments demonstrate that the matching model training on VIS-NIR patch dataset is also suitable for many other muti-modal images,such as optical image and SAR,optical and Li DAR,and can even achieve registration on a sub-pixel level.