| With the development of sensor technology,sensors with various modalities are gradually applied in daily life.Due to the limitations of the sensor,it is difficult for a single-modal image to capture the complete information in a complex scene.Therefore,multiple sensors or imaging devices are of necessity to generate complementary information.Also,the limitations of parameters,such as focal length,aperture,and brightness,cause significant discrepancies between data acquired by the same imaging device or sensor under different shooting conditions.As a technology aiming to enhance the visual effect of images and improve the utilization of effective information,image fusion can detect effective and complementary information from a series of source images that are generated by different modalities or same modality with different parameters,and fuse these effective information into the single image.Apparently,the technology redounds to provide convenience for subsequent computer processing and visual observation.Currently,the image fusion technology has been widely leveraged in many fields,such as computational photography modules in smartphones,medical clinical diagnosis,remote sensing change detection,etc.Generally,multi-source image fusion can be divided into multi-source image fusion and multi-modal image fusion.In recent years,benefiting from CNNs’powerful ability to extract and represent features,various deep learning-based models have dominated various subfields of computer vision.The same goes for the field of image fusion.Various state-of-the-art performance achieved by deep learning-based models demonstrates the fact that deep neural networks can significantly improve the quality of fused images.The reason is that the networks replace the hand-selected image transformation techniques and hand-crafted fusion rules that are indispensable to traditional image fusion algorithms.However,there are some challenges for the deep learning-based models in our field.For example,for supervised models,the lack of real training sets and labels in the field of image fusion is a non-negligible stumbling block;for unsupervised or self-supervised models,explaining the rationality of the chosen pre-training task or pretext task is still an intractable problem.Moreover,existing deep learning models always rely on large-scale training sets,complex loss functions and well-designed structures to achieve breakthroughs in performance,but ignore the prior knowledge and characteristics of image fusion tasks.This study discovers that some problems in the field of deep learning can be solved with the help of traditional theories or techniques.Therefore,on the basis of domestic and foreign research,this study conducts in-depth research on the above problems,taking medical image fusion and multifocus image fusion as sample tasks of multi-modal image fusion and multi-source image fusion respectively.1.Medical Image Fusion Based on Convolutional Neural Networks and Non-Subsampled Contourlet TransformIn the field of medical image fusion(MIF),there are two challenging problems.One is the lack of training sets in the field of MIF.The other is the problem that it is difficult to directly utilize CNNs in the MIF field,which is caused by the pixel intensity differences in different modular images at the same position.To grapple the problems,this study proposes a medical image integration network based on NSCT.The model combines with the advantages of NSCT and CNN to generate promising results.Firstly,multi-modal source images are decomposed by NSCT into low-frequency and high-frequency bands.For high-frequency bands,we design a novel perceptive high-frequency network(PHF-CNN)as adaptive fusion rules.Meanwhile,we propose a new generation method for training sets of PHF-CNN.A significant merit with this method is that training sets and labels are generated from natural images.In terms of low-frequency bands,the strategy of pre-training and transfer learning is exploited for the trained fully convolutional network,facilitating the network with small-scale training sets to generate decision maps.Finally,through the invert NSCT,fused frequency bands are utilized to generate fused images.The experimental results demonstrate that the proposed model outperforms the ten comparison algorithms with state-of-the-art performance.2.A Self-Supervised Residual Feature Learning Model for Multifocus Image FusionTo alleviate the lack of real training sets and labels in the field of multi-focus image fusion,this study proposes a self-supervised residual feature learning model for this task.Our model consists of a pretext task and a fusion module.Through theoretical research and Monte Carlo’s simulatory experimental verification,we discover a residual gradient prior suitable for the image super-resolution task,as well as the multi-focus image fusion task.Consequently,this study selects the image super resolution as a pretext task,transferring the dependence of the neural network to the scarce multi-focus image training set to the natural images.At the training stage,we employ a standard paradigm in the field of image super-resolution to generate a large number of data sets from natural images.Moreover,the proposed prior explains why the image super-resolution can be used as a pretext task for multi-focus image fusion.In the downstream task,the fusion module generates decision maps based on the network’s outputs,i.e.,residual maps.The module consists of an activity level measurement and a new boundary refinement method.Experimental results indicate that the performance of our model is superior to the performance of many comparison algorithms,without any multi-focus image training sets.3.Rethinking the Multifocus Image Fusion:A Model Designed for Essential FeaturesTo solve the problem that characteristics of multi-focus image fusion have yet to be fully utilized and improve the inaccurate boundaries in decision maps,we rethink the task from the perspective of features and thereafter conclude two essential features.Based on the two features,we propose a model consisting of two feature extractors,a feature distillation fusion module(FDFM),and a focus segmentation network Y~UNet.To extract the two essential features of multi-focus images,we select a trained image super-resolution network and a salient object detection network as extractors.The FDFM consists of domain transformation derived from manifold theory and a weight distribution rule.It distills and fuses feature maps from both content and channel dimensions to generate a feature representation suitable for multi-focus images.Y~UNet receives the feature representation as inputs and produces decision maps.The network dynamically adjusts encoding weights for the two essential features and works in the proposed feature space.Experimental results on seven image sets demonstrate that(i)our model outperforms eight competitive comparison algorithms,both qualitatively and quantitatively;(ii)the proposed essential features refine many other MFIF algorithms. |
