Research On Deep Learning-based Multi-modal Image Fusion And Application

In recent years,artificial intelligence technology represented by deep learning has gradually been applied to medical imaging computer-aided diagnosis research and has made progress.However,these studies mostly use single-modal images.Compared with the auxiliary diagnosis system based on a single modality,the multimodal scheme can make full use of the information of different modality,thereby improving the diagnosis level.This dissertation mainly studies the fusion of multi-modal medical images,as well as the fusion of images,gender and age information,and image texture features to distinguish the two types of pancreatic cancer subtypes.At present,CT or MRI images are mostly used in computer-aided diagnosis research related to pancreatic cancer.Based on the advantages of ultrasound examination and imaging,this article uses three modalities of ordinary B-mode ultrasound,arteriography ultrasound and venography ultrasound.The main content of this article is:First of all,this dissertation analyzes and compares the advantages and disadvantages of three multi-modal fusion schemes: data layer fusion,feature layer fusion and decision layer fusion.The data layer fusion has poor versatility and the decision layer fusion accuracy is not high.Therefore,the precision,calculation amount and information are selected.A feature-level fusion scheme with better indicators such as loss.The feature extraction method based on image processing algorithm and the feature extraction method based on convolutional neural network are compared.Since convolutional neural network can automatically extract important features related to classification,convolutional neural network is selected to extract image features.After designing the multi-modal image fusion scheme,the experimental group and the control group are set up.The difference between the experimental group and the control group lies in the selection of different modalities for fusion.Based on the advantages of contrast-enhanced ultrasound specificity and high resolution,the protocol containing the contrast-enhanced ultrasound modalities was set as the experimental group,and the remaining protocols were set as the control group.The experimental results show that,compared with the single-modal image fusion scheme,the fusion scheme using multi-modal images has higher indicators,the accuracy difference between the two is at most 12.5%,and the difference in F1 Score is at most 0.212.In addition,the experimental results also show that the fusion effects of different modes are different.The fusion scheme with ordinary B-mode ultrasound has lower indicators than the fusion scheme without this mode.Secondly,on the basis of the multi-modal image fusion scheme,information such as gender,age and texture characteristics are added.The physiological state of the pancreas varies with gender and age,so the probability of suffering from pancreatic diseases is also different.Gender and age may contain key information to help distinguish different pancreatic diseases.Texture analysis is a commonly used computer-aided diagnosis method that can express the subtle differences between different types of lesions.The extracted multi-modal image features,gender,age,and texture features are fused.The experimental results show that the performance indicators of the scheme are improved,the accuracy rate is increased from 88.5% to91.5%,and the F1 Score is increased from 0.863 to 0.891.Finally,image feature extraction is a crucial part of the multimodal fusion scheme.In order to improve the accuracy of the auxiliary diagnosis model,the structure of the convolutional neural network used to extract image features is improved.The different channels of the extracted image features have different degrees of importance.The channel attention mechanism is introduced in the convolutional neural network to emphasize important features and suppress unimportant features,making it more suitable for the task of image feature extraction.The performance index of the multi-modal fusion scheme of the improved convolutional neural network has been improved.