Infrared And Visible Image Fusion Algorithm Based On Generative Adversarial Network And Attention Mechanism

Infrared and visible image fusion is to combine two images obtained from two different sensors in the same scene to provide a more complex and detailed scene representation.The visible sensor mainly captures the reflected light to obtain the visible image with rich background information,but can be easily affected by environment conditions.The infrared sensor can overcome some obstacles and capture the thermal radiation from the object to obtain the infrared image with obvious contour and fuzzy background.Therefore,the thermal object information in the infrared image and the background information in the visible image are fused into an image,which not only makes it more in line with human visual perception,but also facilitates the subsequent applications of computer vision tasks,such as military surveillance,target recognition and so on.At present,infrared and visible image fusion methods can be roughly divided into two categories: traditional methods and deep learning based methods.Traditional infrared and visible image fusion methods need to manually design the calculation process,and with the improvement of the accuracy and generalization requirements of fusion results,the complexity of the model is also increasing.However,deep learning-based methods usually require a reference image,and the fusion of infrared and visible light images lacks a reference image,and often can only use the source image as a reference image,resulting in a general lack of generalization of the model.Combining deep learning theory with the basis of infrared and visible light image fusion,this thesis conducts in-depth exploration and research on the problems arising from deep and traditional image fusion algorithms,and proposes a new image fusion solution.The main innovative work of the thesis is as follows:1)Taking advantage of the fact that Generative Adversarial Networks do not require reference images,a Generative Adversarial Network and Attention Mechanism-based Network(MAGAN)is proposed,which consists of a multi-attention generator and two multi-attention discriminators.The multi-attention generator is designed as two modules: named tri-path feature pre-fusion module(TFPM)and feature emphasis fusion module(FEFM).TFPM consists of three paths: intensity path,gradient path and pre-fusion path.These paths aim to extract and pre-fuse different features in source images by using the idea of attention.FEFM aims to further emphasize and fuse the output characteristics of TFPM.According to the different characteristics of infrared images and visible light images,two discriminators with different attention mechanisms are constructed,and the adversarial learning with the generator is realized by judging the difference between the fusion image and the source image.In order to retain more complete infrared salient region feature information in the fused image,a new saliency target intensity loss term is designed,which combines adversarial loss,gradient loss and structural similarity loss to train the generator.At the end of this chapter,two ablation experiments are designed to verify the effectiveness of different designs in the model.Experimental results on two public datasets also show that the proposed MAGAN outperforms state-of-the-art models in terms of visual effects and quantitative metrics.2)Aiming at the characteristic that convolutional networks tend to ignore the global dependencies of images,an infrared and visible light image fusion method based on lightweight self-attention transformation module and generative adversarial learning is proposed(TGFusion).It consists of a generator and two discriminators.The generator is an encoder-decoder network and a self-attention transformation module,and the proposed self-attention transformation module can learn the global fusion relationship.The encoder consists of stacks of convolutional blocks to learn shallow features of the source images.Then these shallow features are used as the input of the self-attention transformation module to obtain the global fusion relationship,and then multiplied with them to obtain the optimized features.Finally,these optimized features are used as the input of the decoder,and the fusion result is finally obtained.Besides,adversarial learning provides the generator with different modal features during feature-level training.This is the first attempt of deep integration and application of Transformers and adversarial learning in image fusion tasks.