Research On Infrared And Visible Image Fusion Algorithm Based On Unsupervised Deep Learning

The fusion technology of infrared and visible images is important in the field of image fusion.Infrared images can be used for nighttime vision by detecting the difference of thermal radiation between objects;visible images can provide abundant information of texture,to which the human visual system is sensible.These two images can be effectively fused through image fusion technology.The fused images highlight the thermal targets while maintaining rich texture information,which can help to achieve tasks such as target monitoring and tracking.Traditional image fusion methods are mainly based on multi-scale transformation(MST)and sparse representation(SR).These methods generally feature manual design with much effort,high computational complexity,and low algorithm efficiency.In recent years,methods based on deep learning have gradually been applied to image fusion,and have solved some of those problems.For infrared and visible image fusion tasks,due to no reference fusion image,it is impossible to directly use infrared and visible source images to perform supervised learning.Therefore,most of the existing infrared and visible image deep fusion algorithms are achieved through pre-trained network models or other supervised tasks to complete the training of the network before finishing fusion tasks.However,these methods are not involved in the training of the source image during the network training process,in which the network's adaptive ability is poor.Besides,although these methods have produced good fusion results,they still need to rely on manual design without an end-to-end solution.As such,this research aims to study how to build an unsupervised model for infrared and visible image fusion tasks based on deep learning.The main contributions are as follows:(1)A multi-scale convolutional fusion network(MSC-Fuse)based on DeepFuse is proposed.DeepFuse is a deep auto-encoder network designed for multi-exposure image fusion tasks.The network consists of an encoder,a fusion layer,and a decoder.The encoder is used to extract image features.The fusion layer fuses the deep features extracted by the encoder.The fusion image is finally generated by the decoder.DeepFuse uses a no-reference quality metric to realize unsupervised learning of the network.The main limitation of this algorithm is that the source image features cannot be fully extracted due to the over-simplified encoder network structure.Therefore,a multi-scale convolution module is introduced in this research to effectively improve the feature extraction capability of the network.Besides,a pixel-level loss function is added to the loss function to capture the detailed texture features in the visible image.Experiment results show that this research effectively improves DeepFuse and achieves better fusion results.(2)Inspired by DeepFuse,this research proposes a new unsupervised deep fusion algorithm named UVI-Fuse for the fusion task of infrared images and visible light images,which can directly predict a more informative fused image from source images.The loss function used in this algorithm is the structural similarity index measure(SSIM),determined by combining two no-referenced quality metrics to achieve unsupervised learning.Moreover,the channel attention module is introduced into the network model to further improve fusion performance.Experiment results show that the algorithm can effectively fuse the salient features of infrared images and visible light images.(3)The network structure is further explored,and the fusion algorithm named FE-Fuse based on multi-scale feature enhancement is proposed.In this study,the Rep VGG module is introduced into the encoder.This model can improve the feature extraction capability of the encoder even with only a small number of parameters introduced.Specifically,A multi-scale feature enhancement module is constructed in the feature fusion layer to enhance the spatial dimension of the deep features.Experiment results show that this module can effectively improve the fusion result.In addition,a contrast experiment of commonly used deep feature fusion strategies is carried out to illustrate the basis for the selection of feature fusion strategies.Through experiments,we found that the fusion result of the algorithm FE-Fuse can significantly highlight the brightness information in the source image while it retains the detailed texture information.(4)Because the existing deep fusion algorithms have not considered the multi-scale features of the source image and have not made full use of the deep information extracted by the encoder network,this research tries to improve by constructing a fusion algorithm named Pyramid Fuse based on the image pyramid.Pyramid Fuse first constructs an image pyramid by down-sampling the source image and then inputs the images of different resolutions to their respective encoders to extract features.Finally,the features of different resolutions are fused and reconstructed to obtain the fused image.In the encoder,the algorithm designs multiple branches to the encoder to realize the extraction of the low-level and high-level features of the source image,and designs different fusion strategies for fusion according to the features of different layers.Experiment results show that the algorithm can capture the edge texture information in the source image well and achieve a better fusion effect.