Research On Infrared And Visible Image Registration Methods Based On Two-stream And Modal Transformation

In the wake of developments in modern technology,all kinds of sensors are applied in aspects of life,and people’s requirements for data acquisition are also increasing.However,the data collected by a single sensor has limitations to a certain extent.Visible light sensors and infrared sensors are common complementary image sources in daily life.Infrared images can be imaged based on the thermal radiation of the object,which are not affected by light,but generally have low resolution and fuzzy imaging.Visible images contain rich color and texture details but are heavily affected by light.The fusion of infrared images with visible images will bring great convenience to human life.However,infrared images and visible images are not aligned in most cases,so image registration is required before image fusion.Image registration refers to mapping one image to another image by finding an appropriate spatial transformation relationship between two images corresponding to different imaging conditions.As an important prerequisite before image fusion,the accuracy of image registration will directly affect the quality of fused images.In this paper,based on the deep learning algorithm,the characteristics of two spectral images are studied and two infrared and visible image registration network models are proposed,which work as follows.(1)In this paper,TEIVRNet,a two-stream infrared and low-light visible image registration network fused with edge information is proposed.Firstly,the network uses Random Color Jittering to enhance low-light visible images to obtain the enhanced images.The enhanced images and infrared images form one structural stream,and the low-light visible images and infrared images form another structural stream,forming a two-stream structure,which alleviates the problem of incomplete feature expression of the low-light visible image.Then,the DO-CARes module,which is mainly composed of Depthwise Over-parameterized Convolutional Layer(DO-Conv)and Coordinate Attention(CA),proposed in this paper is used to enhance the feature extraction capability of the network.Finally,the Richer Convolutional Features for Edge Detection(RCF)network is used to complement the edge information of the infrared images.So,the TEIVRNet can better obtain the common features between the infrared images and the low-light visible images.The experimental results on the infrared and low-light visible image dataset LLVIP show that the Probability of Correct Keypoints(PCK)of TEIVRNet is 22.15% higher than the original basic network when = 0.1.At the same time,compared with other methods,the visual registration effect is better.(2)In this paper,MTIVRNet,an infrared and visible image registration network based on modal transformation is proposed.Firstly,Cycle Generative Adversarial Network(CycleGAN)is used in MTIVRNet to train infrared images and visible images to transform the visible images into pseudo-infrared images,which reduces the spectral difference between the two images.Then,in the feature extraction stage,the OD-Dense module proposed in this paper is used,which is mainly composed of Omni-dimensional Dynamic Convolution(ODConv),which can effectively improve the feature extraction ability of the network while reducing the number of parameters.The experimental results on the infrared and visible image dataset FLIR show that the proposed registration network has a better performance compared with other methods.The experimental results on infrared and visible image dataset FLIR show that the PCK of MTIVRNet is 9.48% higher than the original basic network when = 0.1,and also shows better visual registration effect.