Deep Learning Based Methods And Applications On Multi-modal Medical Image Analysis

Background and Objectives:In recent years,artificial intelligence-based deep learning techniques have been applied in the field of medical image processing and assisted diagnosis.By using deep learning algorithms,image processing and lesion analysis models with high levels of automation can be trained for medical imaging,enabling fast and accurate analysis and diagnosis of various types of diseases.The application of this technology can greatly improve the efficiency and accuracy of doctors,while reducing medical errors and unnecessary medical treatment costs for patients.However,intelligent analysis of multi-modal medical image data remains a challenging task.Multi-modal medical imaging refers to the use of different imaging techniques,imaging sequences,and imaging perspectives to examine the same patient,in order to provide more comprehensive and accurate diagnostic information.Multi-modal medical image data has a vast and complex scale,and each modality of images has unique features and information,including different imaging parameters,different perspectives,and different physical meanings.In the process of training deep learning models,one of the current research challenges is how to use prior knowledge of human anatomy,disease etiology,or imaging principles to guide the training and analysis of deep learning models,enhance training efficiency,and prevent overfitting.Additionally,multi-modal medical images often have more severe noise and interference,and the patient’s angle,position,and posture can vary during imaging,making intelligent analysis more challenging.Finally,compared to natural images,medical images suffer from sampling and class imbalance issues,which pose obstacles to the cross-domain application of deep learning methods.This research aims to address the challenges in information acquisition,mining,fusion,and interpretation of multi-modal medical images,as well as explore overfitting phenomena and unlabeled evaluation methods in the cross-domain application of deep learning.To achieve this goal,this study focuses on three specific aspects.Firstly,for the discrimination and detection of benign and malignant breast cancer,the study investigates deep learning feature extraction and combination methods on multiparameter breast MRI images and explores their utility in tumor malignancy classification and detection.Secondly,the study explores how to use deep learning models for automatic identification,classification of vessel stenosis degree,and real-time detection of angiographic images under multi-angle Coronary Angio Gram(CAG)imaging conditions.Lastly,the study investigates unlabeled training and overfitting issues in multi-modal image classification and segmentation tasks under imbalanced sample and unlabeled conditions and applies this method to specific medical image processing tasks.Through the aforementioned research tasks,this paper aims to contribute to the development of the interdisciplinary field combining medical imaging and artificial intelligence.On one hand,it provides computer-based methods for the interpretation and analysis of multi-modal images in the field of medical imaging,offering more convenient,fast,and accurate clinical application support.On the other hand,it delves into the mathematical laws hidden behind the data,addresses common scientific problems encountered in the integration of medicine and engineering,and provides a novel approach to solving problems in deep learning methods for multi-modal medical imaging.Materials and methodsPart Ⅰ: MRI-based breast cancer classification and localization by multiparametric feature extraction and combination using deep learning1.Study subjects: The internal cohort consisted of 569 patients who underwent multimodal breast MRI examinations at Daping Hospital of the Chinese People’s Liberation Army Military Medical University between October 2016 and December 2020.Additionally,an external cohort comprising 125 breast multiparametric MRIs from the TCGA-BRCA dataset was included in the analysis.These datasets provided a diverse and representative sample of patients for evaluation.2.Image acquisition: Image acquisition for the breast MRI examinations involved the use of a 1.5T magnetic resonance imaging scanner.T1-weighted imaging and dynamic contrast-enhanced MRI(DCE-MRI)were performed using gradient echo sequences,while T2-weighted imaging(T2WI)utilized spin echo sequences.Diffusion-weighted imaging was conducted using single echo plane sequences.The multimodal nature of the imaging data allowed for a comprehensive analysis of the breast tissue,capturing different aspects of the pathology.3.Image processing: For image processing,a convolutional neural network(CNN)combined with a short-term memory network(LSTM)was employed to extract features and classify the lesions present in the multimodal MRI data.Histopathology served as the reference label for categorizing lesions as malignant or benign,while the contralateral healthy breasts in the internal and external cohorts served as a control group.The class activation mapping(CAM)method was employed to localize the lesions within the breast tissue.In the internal cohort,the performance of the deep learning model was compared to the evaluations of three independent radiologists using the BI-RADS method as a reference.4.Statistical analysis: To evaluate the classification performance of the lesions,various statistical measures were utilized,including sensitivity,specificity,area under the curve(AUC),De Long test,and Cohen’s kappa.These metrics provided a comprehensive assessment of the model’s ability to accurately distinguish between malignant and benign lesions.Additionally,the localization performance was evaluated using sensitivity and mean square error(MSE),which measured the model’s ability to accurately pinpoint the location of the lesions within the breast tissue.Statistical significance was determined using a P-value threshold of <0.05.Part Ⅱ: A deep learning method for vascular stenosis detection in multi angle coronary angiography images1.Study subjects: In this study,a deep learning method was developed to detect vascular stenosis in multi-angle coronary angiography(CAG)images.The analysis involved a retrospective analysis of 230 patients with multi-ethnic and multi-center CAG data obtained from the CORE 320 dataset.The study population had a median age of 62 years [IQR 55,69],with 70% male representation.Among the participants,45% were white,82% had hypertension,71% had dyslipidemia,16% were current smokers,and 27% had obstructive coronary artery disease.This diverse patient population provided a comprehensive dataset for the evaluation of the proposed deep learning method.2.Image acquisition: For image acquisition,a total of 230 cases comprising 13,744 frames of CAG images were used for training and validation of vascular stenosis classification.The coronary angiography images included both left coronary artery(LCA)and right coronary artery(RCA)views.The multi-angle view consisted of four angles for LCA(LAOCRA,LAOCAU,RAOCRA,and RAOCAU)and three angles for RCA(LAO,RAO,and CRA).This wide range of angles and views allowed for a comprehensive evaluation of vascular stenosis in different regions of the coronary arteries.3.Image processing: The image processing stage involved the development of a multiangle stenosis detection framework based on deep learning techniques.Firstly,a combination of convolutional neural networks(CNN)and long short-term memory(LSTM)networks was utilized to select candidate frames from the CAG video sequence.This initial step aimed to identify potentially relevant frames for further analysis.Next,Inception V3,a popular deep learning architecture,was employed to classify the stenosis in single artery single angle images,with redundant training techniques implemented to mitigate overfitting.Subsequently,image features from different angles were fused,enabling stenosis classification at both the arterial and patient levels.Finally,the class activation mapping(CAM)and Feature Pyramid Network(FPN)methods were applied to locate the stenosis within the images.4.Statistical analysis: The statistical analysis of the results encompassed the classification and localization of stenosis at the image,artery,and patient levels.Various metrics were employed,including accuracy,F1 score,Cohen’s Kappa,and area under the curve(AUC),to evaluate the classification performance of the stenosis detection method.Stenosis localization was assessed using sensitivity,specificity,and mean square error(MSE).To ensure robust evaluation,a 4-fold cross-validation strategy was employed.All statistical analyses were conducted using Python and the Scikit-learn library.Continuous variables with a normal distribution were summarized and reported as mean ± standard deviation,providing a comprehensive summary of the quantitative results.Part Ⅲ : An unlabeled evaluation and ranking method for convolutional neural networks1.Research object: This study assumes that the performance of CNN models in image classification is determined by the diversity of active feature patterns.The method for calculating its diversity is the normalized entropy of the weighted activation histogram of the convolution kernel,which is defined as the activation entropy.Subsequently,in the classification and segmentation tasks of natural images and medical images,the correlation between the quantitative evaluation of labeled and unlabeled performance of different CNN is compared,and the method is applied to the automatic verification process of unlabeled CNN to prove the hypothesis.2.Materials: Natural image classification dataset from CIFAR10 and Image Net,thoracic and abdominal CT classification dataset from Deep Lesion,pathological image classification dataset from Med MNIST,natural image segmentation dataset from VOC2012,breast multimodal MRI ataset from Part I,and CAG multi-angle vascular stenosis classification dataset from Part II.3.Experimental Design: In natural image and medical image classification tasks,100 CNN with different hyperparametric settings are trained based on six classification models,and their activation entropy values,and label-based classification accuracy values are calculated.In the segmentation task,100 CNN with different hyperparametric settings are trained based on the two segmentation models,and the mean intersection and union(m Io U)of their activation entropy and label-based segmentation is calculated.In the unlabeled automatic verification task,activation entropy is used instead of verifying accuracy to participate in the model optimization of the training process,and compared with the optimal model with label verification.Finally,the activation entropy method is used to optimize the CNN classification model in MRI breast cancer classification task and CAG vascular stenosis classification task,and its effects are compared.4.Statistical analysis: Pearson correlation coefficient,Spearman rank,and Kendall rank were used to measure the correlation between labeled and unlabeled methods.Top-1 accuracy was used to measure model classification performance in unlabeled automatic verification tasks,and the root mean square error(RMSE)of accuracy was used to measure the performance gap between the unlabeled and the labeled validation method.Results:Part Ⅰ:MRI-based breast cancer classification and localization by multiparametric feature extraction and combination using deep learning1.In an internal test queue consisting of 278 samples,the optimized combination of multiparametric MRI(mp MRI)achieved impressive results.The deep learning method attained an area under the curve(AUC)of 0.98 and a sensitivity of 0.96 in the task of classifying tumor benignancy and malignancy.Notably,even without dynamic contrastenhanced MRI(DCE-MRI),the deep learning method surpassed the readings of radiologists,yielding an AUC of 0.96 compared to the radiologists’ score of 0.90.2.The evaluation further extended to an external queue,which comprised 125 samples.In this setting,the deep learning method achieved an AUC of 0.91 and a sensitivity of 0.83.Interestingly,by incorporating 10% of the external image data into the internal training set,the performance of the deep learning method showed substantial improvement.The AUC rose to 0.94,while the sensitivity increased to 0.89.3.Furthermore,the study investigated the effectiveness of using DCE-MRI or T2 WI sequences individually for tumor detection in the images.The localization sensitivity obtained from DCE-MRI was 0.97,while T2 WI achieved a sensitivity of 0.93.These results highlight the potential of deep learning methods in accurately detecting breast cancer lesions using different MRI sequences.The promising findings from this part of the study demonstrate the efficacy of deep learning techniques in extracting lesion features and classifying breast cancer in multimodal MRI.These advancements have the potential to enhance the accuracy and efficiency of breast cancer diagnosis,ultimately leading to improved patient outcomes.Part Ⅱ: A deep learning method for vascular stenosis detection in multi angle coronary angiography images1.To evaluate the detection of contrasting frames in CAG,a total of 582 videos from 175 patients were examined.The deep learning method achieved an acceptance rate of 83% and an error rate of 5.0%.Moreover,the detection error for the start and end frames of the imaging process was measured at 2.05 and 2.27,respectively.These results indicate the efficacy of the deep learning method in accurately identifying contrasting frames in CAG images,which is crucial for subsequent analysis and diagnosis.2.The stenosis classification evaluation was conducted using 13,744 images from 230 patients.The severity distribution of the stenosis cases included 46 cases(20%)with stenosis below 25%,127 cases(55.2%)with stenosis between 25% and 99%,and 57 cases(24.8%)with complete stenosis(100%).The deep learning method demonstrated excellent performance in classifying vascular stenosis at the image level.For the classification of stenosis on the left coronary artery(LCA)and right coronary artery(RCA)based on the "<25%" and ">25%" thresholds,the method achieved accuracies of 0.77 and 0.84,respectively.The sensitivity values were 0.85 for LCA and 0.86 for RCA.Additionally,the method achieved high accuracy in stenosis prediction at the artery level,with accuracies of0.82 for LCA,0.86 for RCA,and 0.85 for patient-level predictions.The corresponding sensitivity values were 0.94 for LCA,0.92 for RCA,and 0.95 for patient-level predictions.These results demonstrate the effectiveness of the deep learning method in accurately classifying and predicting the severity of vascular stenosis in CAG images.3.The localization evaluation of vascular stenosis focused on 690 images,accompanied by 1588 manual annotations as reference criteria.Two positioning methods,CAM(class activation mapping)and FPN(feature pyramid network),were employed for the evaluation.The CAM-based positioning method achieved sensitivities of 0.59 for LCA and 0.61 for RCA,with mean squared error(MSE)values of 103.3 for LCA and 79.5 for RCA.On the other hand,the FPN-based positioning method achieved higher sensitivities of 0.68 for LCA and 0.70 for RCA,with reduced MSE values of 39.3 for LCA and 37.6 for RCA.These findings indicate that the deep learning-based positioning methods can effectively localize vascular stenosis in CAG images,with the FPN-based approach demonstrating superior performance.The results obtained in this part of the study underscore the potential of deep learning methods for accurate detection,classification,and localization of vascular stenosis in multiangle CAG images.By leveraging the power of artificial intelligence,these methods can assist medical professionals in diagnosing and treating cardiovascular conditions more effectively.The findings contribute to the field of medical image processing and provide valuable insights for the development of advanced computer-aided diagnostic tools.Part Ⅲ: An unlabeled evaluation and ranking method for convolutional neural networks1.To validate the proposed activation entropy method,several validation sets were utilized,including Image Net,Cifar10,Med MNIST,Deep Lesion,and VOC2012.The results demonstrated a significant correlation between activation entropy and classification accuracy across eight different CNN structures.The Pearson coefficients for the correlation were measured at 0.881 for Image Net,0.756 for Cifar10,0.907 for Med MNIST,0.668 for Deep Lesion,and 0.717 for VOC2012.These findings indicate that the activation entropy metric can serve as a reliable indicator of classification accuracy across diverse datasets and CNN architectures.2.In the tasks of model training for automatic verification without tags,the accuracy and m Io U(mean intersection over union)metrics were compared between the activation entropy evaluation and labeled evaluation methods.The results showed that the activation entropy method outperformed the other two commonly used unlabeled evaluation methods in all four tasks.Specifically,the accuracy difference between the activation entropy evaluation and the labeled evaluation methods was measured at-1.2%,while the m Io U difference was-0.011.These results suggest that the activation entropy method provides more accurate and reliable evaluations of CNN models trained without labeled data.3.Furthermore,the performance of classification models optimized using the activation entropy method was assessed in two specific tasks: breast cancer classification using 1-3sequences and CAG stenosis classification.In the breast cancer classification task,the models optimized with activation entropy exhibited improved performance compared to the baseline models.The improvements in accuracy(ΔAcc)were measured at 0.016,0.026,and 0.024 for1,2,and 3 sequences,respectively.Additionally,the area under the curve(AUC)values showed significant improvements(ΔAUC)of 0.061,0.033,and 0.021 for 1,2,and 3sequences,respectively.In the CAG stenosis classification task,the models optimized using activation entropy achieved higher accuracy and AUC values compared to the other two unlabeled evaluation methods.Specifically,the accuracy values reached 0.69 and 0.81,while the AUC values were 0.80 and 0.85 for LCA and RCA,respectively.These results indicate that the activation entropy method enables more effective optimization of models for CAG stenosis classification,leading to improved accuracy in identifying and classifying vascular stenosis in coronary angiography images.Overall,the findings of this part of the study highlight the utility and effectiveness of the proposed unlabeled evaluation and ranking method for CNNs.The activation entropy metric serves as a reliable indicator of classification accuracy across different datasets and CNN architectures.By leveraging this method,the performance of CNN models can be significantly enhanced in various tasks,including breast cancer classification and CAG stenosis classification.These advancements contribute to the development of more accurate and efficient deep learning models for medical image analysis and diagnosis.Conclusion:1.A multimodal MRI breast tumor detection method based on deep learning achieves high-precision lesion detection in both internal and external queues through feature extraction and combined classification of multiple sequence MRI images.Applying this method to a non-contrast agent sequence combination can achieve AUC and sensitivity comparable to those obtained by radiologists on DCE-MRI.2.Deep learning methods can accurately detect arterial stenosis in coronary angiography images.This method eliminates the steps of vessel extraction and segmentation in coronary artery stenosis classification and CAG image localization and improves the accuracy of vessel stenosis detection at the arterial and patient levels through a framework of multi-angle joint classification decision-making.3.Activation entropy can quantitatively measure the diversity of feature activation in CNN without relying on tags or specific network structures.Experimental results show that activation entropy can be seen as a non-label alternative to label evaluation and verification in supervised learning,improving the classification performance of CNN on non-label datasets.In summary,the application of deep learning-based multi-modal medical image processing methods in fields such as breast cancer and coronary angiography demonstrates significant superiority and potential.Additionally,the proposed activation entropy evaluation method offers a new approach for assessing deep learning models on unlabeled datasets,which is expected to play an important role in medical image processing and other interdisciplinary applications.These research findings will drive the organic integration of deep learning and medical imaging,providing strong support and assurance for improving medical diagnostic accuracy and efficiency,exploring new diagnostic and therapeutic methods,and offering valuable insights.