Research And Application Of Multi-modal Medical Image Fusion Based On Deep Learning

With the rapid development of medical imaging technology,medical images collected by different sensors and containing different pathological information in the clinic,that is,multimodal medical images,have become indispensable tools for analyzing and diagnosing diseases.Because a single modality image cannot provide comprehensive pathological information for disease diagnosis,doctors often need to observe multiple images of different modalities at the same time in clinical practice.However,this is not only difficult to observe but also has visual errors,which greatly increases the uncertainty of diagnosis.Multimodal medical image fusion technology can combine pathological information from multiple images into one image,which not only utilizes the advantages of different modality images but also compensates for the shortcomings of a single image,providing a more reasonable and reliable basis for disease diagnosis.In recent years,researchers have proposed various advanced image fusion theories.Multimodal medical image fusion has achieved good results,but there are still some shortcomings and challenges.Firstly,in terms of the quality of the fused image,existing fusion methods often suffer from problems such as color distortion,contrast deviation,blurred edges and details,and the introduction of noise.Secondly,many methods design increasingly complex algorithm models to obtain better fusion effects,leading to a significant increase in fusion time and an inability to meet the requirements for fusion efficiency in specific scenarios.Thirdly,due to the feature distribution differences between multimodal medical images,which are usually not considered or insufficiently addressed in existing algorithms,important feature information is lost during the fusion process.This thesis provides a detailed analysis of the objectives of multimodal medical image fusion and proposes innovative improvements to address the aforementioned shortcomings and challenges.The main contributions of this thesis are as follows:(1)A fusion algorithm based on rolling guidance filtering and convolutional neural network is proposed to address the existing problems of blurred edges,insufficient details,and noise introduction in existing fusion algorithms.This algorithm decomposes the source images into base and detail sub-images using rolling guidance filtering and further preserves the detailed information in the detail sub-images by employing a fusion rule based on summodified-Laplacian.Additionally,a convolutional neural network is designed to extract and adaptively integrate structural features from the base sub-images,resulting in a fused base subimage with clear edge structures.Finally,the fused sub-images are reconstructed to obtain the fusion image.This algorithm effectively solves the problem of insufficient preservation of small-scale information such as edges and details in the fusion image,and mitigates the impact of noise in the source images on the fusion performance.(2)Aiming at the problem of low fusion efficiency in work(1),and to further solve the problems of color distortion,contrast deviation,and feature distribution difference,this thesis proposes a fusion algorithm based on saliency perception and generative adversarial network.Firstly,a hierarchical and progressive generative network is designed to adaptively extract and fuse multimodal features for generating a fusion image,fully utilizing the correlated and complementary information among multimodal images.Secondly,a saliency awareness strategy is proposed to evaluate the saliency of source images,guiding the generator network to extract salient features in the scene.In addition,an adversarial learning method based on decomposition consistency is designed to improve the quality of fusion images while addressing feature distribution differences.The proposed algorithm effectively alleviates the performance degradation caused by feature distribution differences in image fusion,and the generated fusion image is perceptually consistent with the human visual system.The excellent fusion efficiency meets the real-time requirements in practical clinical applications.(3)To apply image fusion technology to clinical practice,a multimodal medical image fusion system was designed and implemented based on two image fusion algorithms proposed in this thesis.This system provides various functions such as image reading,preprocessing,image registration,image fusion,and image saving.The intuitive interface and user-friendly interactive design of the system help doctors efficiently analyze the condition of the patient and develop treatment plans.