Multi-sensor Deep Image Fusion Based On Deep Learning

Image fusion is designed to combine images acquired by multiple sensors to generate an image with more information and more robustness.With the widespread application of deep learning in computer vision tasks,image fusion based on deep learning has also been greatly developed and achieved excellent performance.This thesis focuses on multi-focus image fusion and the infrared and visible image fusion based on deep learning.At the beginning,some image fusion methods are summarized,especially,three methods which are closely related to this thesis are discussed in detail.Based on this,two kinds of image fusion methods are proposed.For multi-focus image fusion,how to efficiently extract and use detail information and context information,and how to achieve accurate fusion and reconstruction between features play an important role.To this end,a multi-focus image fusion algorithm based on context information and detail injection is proposed.The acquisition of detail information directly affects the quality of the fusion result,and the context information can help the network to identify the focus area.Therefore,in order to extract rich detail and context information,a detail and context information extraction module is proposed.Based on this,a feature extraction network is created to extract multi-level image features.And then,a fusion injection module is designed to achieve accurate feature fusion and reconstruction from coarse to fine.Finally,the fused image is obtained through the reconstruction module.In addition,since there is no public training data set for this task,a new and more reasonable training data set is created.Aiming at the problem that the infrared and visible image fusion task does not have the ground truth,an infrared and visible image fusion algorithm based on self-supervised learning is proposed.First,a pre-trained network is used to generate an initial pseudo ground truth image.And then,it is combined with infrared and visible images to participate in the learning of the network together.In order to realize the optimization of the pseudo ground truth image,the fused image generated when the network converges is used to update the pseudo ground truth image.Finally,after three stages of learning,the network finally outputs the optimal fusion result.During feature extraction,an improved pre-trained VGG16 network is used as the basic network to extract multi-level image features that contain rich detail information and context information in a manner of fusing adjacent features.In the process of feature fusion,a weight normalization fusion module is proposed to complete the feature fusion between different modalities according to the characteristics of the data itself,which avoids the difficulties caused by artificially designed fusion rules.In addition,in order to realize the accurate learning of the network,a joint loss function is proposed,which consists of pixel-level loss,image block-level loss,and feature-level loss.Finally,the two proposed deep-learning-based image fusion methods are tested on publicly available data sets,and compared with existing mainstream algorithms.The proposed algorithm achieves satisfactory fusion performance regardless of visual quality and objective assessment.