Research On Multi-modal Medical Image Fusion Method Based On Deep Learning

In recent years,with the improvement of medical technology,the medical imaging technology has made great breakthroughs,and plays an extremely important role in clinical medicine.However,due to different imaging principles,the content and information contained in the images generated by different imaging devices are various.Doctors need to read a large number of medical images with different modalities if they want to obtain the comprehensive information of the lesions.For doctors’ brain and physical strength,the huge workload is a serious challenge.Misdiagnosis and missed diagnosis are sometimes inevitable.In light of this,multi-modal medical image fusion technology came into being.Medical image fusion technology can extract the unique salient features of medical images with various modalities based on the spatiotemporal correlation and information complementarity,and then fuse these features into one image.It can greatly reduce the number of films that doctors need to read,and effectively improve the efficiency of disease diagnosis and treatment.At the same time,with the rapid development of deep learning,research in this field has also made great progress.However,when images have more texture details or uneven feature distribution,most fusion methods cannot fully and evenly fuse the features with different modalities,which are usually displayed as the lack of the fused images’ texture features,and unbalanced feature performance in the fused images.In view of the above problems,this paper focuses on the preservation of texture details and the balanced feature fusion.The main works of this paper are as follows:Firstly,multi-modal medical image fusion is an unsupervised task,lacking ground-truth as a reference for fusion results,thus unable to constrain the fusion process effectively.At present,most methods mainly apply artificial priori to design fusion constraints,which is easy to cause that the fusion process are unable to fully extract and retain complex features from the source images.To solve this problem,a multi-modal medical image fusion network based on symmetric dual-adversarial constraints is proposed,which attempts to constrain the fusion process through adversary,and then to improve the fusion effect.The network includes an improved generator module based on U-Net and two symmetrical discriminator modules.The generator aims to extract and fuse the features with different modalities.The purpose of the discriminators is to distinguish the fused image from the source images as far as possible.The generator and discriminator are trained for learning from each other to estimate the potential distribution of sample features and improve fused images’ visual performance.The evaluation results on three multi-modal medical image datasets show that the fused image generated by this method has improved the visual effect and balanced feature retention significantly.Secondly,the number of samples in the existing registered multi-modal medical image datasets is generally small.When the network structure is too complex or the parameters are too many,the small dataset is not enough to train a model which has powerful feature representation and feature extraction capabilities.However,the simple network cannot accurately capture the distribution of sample featuras.In view of this,it is proposed to integrate the pre-trained feature extraction model into the multi-modal medical image fusion network with symmetric dual-adversarial constraints.This network consists of a pre-trained feature extraction module,a feature fusion module and two symmetric discriminators.The training process of this network is the same as that of the above method,but the difference is that the features need to be fused are extracted by the pre-trained model.In addition,the parameters of the feature extraction module must be fine-tuned during the training process to make it more suitable for the downstream tasks.The experimental results on the above three datasets show that the pre-trained feature extraction model makes the texture as well as other detailed features in the fused images clearer,and makes the visual effect further improved.Thirdly,due to the complexity and instability of the adversarial training process,a convolutional attention fusion method based on multi-scale features is proposed,which aims to extract multi-scale features from the source images by deep neural network,and then fuse them.The network includes high/low frequency feature extraction module,same-scale feature fusion module,and fused image reconstruction module.Among them,the extraction of high-frequency features is performed by the pre-trained convolutional neural networks(CNN)model,and the extraction of low-frequency features is completed by multiple down-sampling;The same-scale feature fusion module is composed of the convolutional block attention module(CBAM),which attempts to learn and strengthen salient features by calculating the channel attention and spatial attention respectively.The experimental results show that fusing the multi-scale feature from source images can effectively preserve the image edges and texture details.