Research On Infrared And Visible Image Fusion Based On Unsupervised Deep Learning

Infrared and visible image fusion technology is one of the research hotspots in the field of image fusion.The infrared image reflects the thermal radiation information of the object and has strong indication characteristics to the hidden target.By combining it with the visible image with high resolution and clear scene texture,it can highlight the target while keeping the background details,which is conducive to the all-weather work of the system to achieve target detection,tracking,monitoring and other tasks.The traditional infrared and visible image fusion method developed earlier and has been widely used.In the process of image fusion,although the transformation mode is flexible,the activity level measurement and fusion rules still need to be set manually,which is relatively difficult to realize and the model is difficult to adjust.In recent years,deep learning has been applied to infrared and visible image fusion and achieved good results.By training the multi-layer and deep neural network to learn the complex relationship between the data,the image feature information can be automatically extracted and then fused under the constraint of loss function,which makes the model more concise,intelligent,easy to adjust and highly adaptable.In the dataset of infrared and visible images,the source images are insufficient,and there are no standard reference images for fusion.Currently,several deep learning fusion methods are applied,most of which require pretraining on other visible light datasets,and the training is not completely consistent with the test model,so the end-to-end direct mapping of input source image to output source image cannot be realized.The implementation processes of these supervised networks are complex and the generalization ability of the fusion models is poor because the infrared and visible source images are not involved in the training of the network.In view of the above problems,this paper proposes and designs two end-to-end fusion models based on the amplification of the original dataset from the perspective of unsupervised deep learning.No pre-training is required,and the fusion efficiency and quality are improved.The experiment results prove the satisfactory performance of the proposed method.The main research contents of the paper can be summarized as the following six points:(1)This paper systematically introduces the components and characteristics of network models commonly used in deep learning,including convolutional neural network(CNN),residual network(ResNet),densely connected convolutional network(DenseNet)and generative adversarial networks(GAN).The definitions of convolution,pooling,loss function and back propagation in the training of deep neural network are explained.The fusion method based on the above networks and the traditional fusion methods are described to lay a foundation for the related research in this paper.(2)The infrared and visible image fusion model based on unsupervised convolutional neural network is studied.The construction of network architecture is very important for the realization of fusion.In this method,DenseNet is introduced as a feature extraction and transfer sub-network,which makes full use of the features of each layer and connects its channels for feature reconstruction of the fused image,thereby ensuring that the decoded fused image contains multi-scale and multi-level features of the source image.The significant target and detail information in the source images are effectively preserved and the image quality is improved.(3)The application of perceptual loss in the training of infrared and visible image fusion is studied.By reducing the difference between higher-level features extracted by the perception network on real image and the generated image,the generated image can be continuously optimized and close to the target image gradually,which has a better effect in the super-resolution reconstruction and image style migration.It can be applied to constrain the difference of perception features between the source images and the fused image,so that the fused image can preserve more and more information of source images,and the fused image is easier to be observed by human visual system and has better visual effect.(4)A fusion model based on generative adversarial network and residual network is studied.The fused image including infrared intensity and visible gradient information is generated by the generator,and other details of the visible image are added to the fusion image gradually by the discriminator.In the original fusion model,the generator model and loss function are relatively simple.In this paper,the residual network is used as the feature extractor in the generator network,combined with additional skip connections,to ensure that the features of the front layer can be retained and the residual information learned is helpful for the establishment of the fused image.The structural similarity loss is introduced to constrain the fusion image and the source images on the whole structure.(5)Experimental verification of fused images.The experimental environment was built on the GPU platform,and the network architecture,loss function design and the whole training and testing process were realized under the TensorFlow framework based on Python language coding.Through subjective and objective evaluation,the test results of the two methods in this paper were compared with the traditional methods to verify the feasibility of the methods.At the same time,the key designs in each model are analyzed independently to verify their effectiveness.(6)The fusion results of this method are evaluated objectively and comprehensively.In the paper,definitions of subjective and objective evaluation methods are introduced and some commonly used objective evaluation indicators are explained in detail.On the basis of referring to other evaluation systems and quantitative results of objective evaluation indexes,this paper establishes an objective comprehensive evaluation system from two aspects: the fusion image itself and the correlation between the fusion image and the source images.Through MATLAB software,the fusion results of this method and other 7 traditional and deep learning methods were evaluated,and the fusion results of this method were compared and analyzed.