Research On Image Fusion Methods Based On Deep Learning

With the development of science and technology,a large number of image fusion methods have been proposed.In the early stage,some image fusion methods based on multi-scale transform aim to transform the images into frequency domain for calculation and fuse the coefficient in frequency domain,and then reconstruct the fusion result through the inverse transformation.These methods greatly increase the computation of the algorithms.Therefore,some researchers proposed to directly process the source images and extract image features,such as sparse representation and low-rank representation.The disadvantage of the representation learning-based methods is that it needs to learn an overcomplete dictionary to represent the input images which will make the overall operation efficiency of the algorithm very low.Therefore,in recent years,some researchers have tried to find a simpler method of feature extraction and fusion,and the method based on deep learning can meet these requirements to some extent.A large amount of training data is used to train an end-to-end network,and then the trained network is used to extract features of source images and fuse them to generate the fusion result.Meanwhile,inspired by image fusion methods based on deep learning,we propose an image fusion method based on an improved multi-level decomposition method.Therefore,this paper mainly focuses on the image fusion methods based on deep learning.The main contributions of this paper are as follows:?1?The training time of convolutional neural network?CNN?is very time-consuming and complex.Therefore,this paper proposes to apply a very simple deep learning model,called PCANet,to multi-focus image fusion.We utilize the pretrained PCA filters to extract image features of source images.Then,the activity level maps are generated by using nuclear norm.After that,we leverage a series of post-processing operations to process them to generate the final decision map.Finally,the fusion result is achieved accordingly.The comparison with other methods shows that the method has achieved better fusion performance in subjective and objective evaluation.?2?Based on the fusion framework proposed by Dense Fuse,from the perspective of extracting multi-scale features,the multi-scale feature layer is added into encoder and the fusion result is reconstructed by richer feature representation.Therefore,we propose multi-scale DenseNet?MSDNet?and apply it to medical image fusion.The main framework of our fusion network consists of encoder,fusion layer and decoder.Firstly,the multi-scale features of the source images are obtained by encoder,in fusion layer,the fusion strategy based on l₁-norm spatial attention to fuse multi-scale features respectively.Finally,the fused image is reconstructed by decoder.Compared with other methods,the proposed method has achieved better fusion performance in objective and subjective evaluation.?3?We continue to research the fusion method based on multi-scale features.We propose a Res2Net-based infrared and visible image fusion method.Our fusion model consists of three parts:encoder,the fusion layer and decoder.Moreover,we propose a new training strategy which only use a single natural image to train a Res2Net-based reconstruction model.The insight on the proposed method is analyzed.Firstly,we used Res2Net-based encoder to extract the multi-scale features of the source images.The fused features will be obtained by the fusion strategy based on spatial attention model.Finally,the fusion result is reconstructed by the decoder.Experimental results show that our method has achieved better fusion performance from objective and subjective evaluation.?4?Inspired by deep learning-based image fusion methods and a novel multi-level decomposition method?MDLat LRR?,we propose a medical image fusion method based on an improved image decomposition method?MDLatLRRv2?.Because MDLat LRR pays little attention to low-frequency information which is obtained from the latent low-rank representation?Lat LRR?and not effectively used.Therefore,we propose to learn a new low-rank projection matrix to replace the operation of extracting low-frequency information in Lat LRR,at the same time,effectively avoid the limitation on the size of the input image.So the fusion algorithm can handle any size of the input image.We apply the MDLat LRRv2 to the medical image fusion.A nuclear-norm based fusion strategy is used to fuse the detail parts and the base parts are fused by an averaging strategy.Compared with other methods,the experimental results show that the fusion performance is better from both subjective and objective perspectives.