Research On Prognosis Prediction Method Of Rectal Cancer Based On Multi-Instance Learning Of Medical Images

Medical images are widely used in the diagnosis and treatment of cancer patients,and are usually presented as a collection of multiple sub-regions or multiple components.For example,magnetic resonance imaging(MRI)consists of grayscale images with different parameters and layers,and whole slide images(WSIs)can be divided into pathological patches representing different tissue areas.The components of these medical images contain a large amount of patient prognostic information.Multiple-instance learning(MIL)is a special paradigm under the weakly supervised learning framework.It aims to construct an efficient mapping between image component(instance-level)information and the overall image label through artificial intelligence analysis.Therefore,it fits the patient prognosis prediction paradigm based on medical image analysis.Mortality rates among young patients due to rectal cancer are increasing year by year.Accurate prognosis prediction of rectal cancer patients is expected to assist clinicians to develop personalized treatment plans and benefit patients.As routine clinical tests,noninvasive MRI and gigapixel WSI can reflect patient prognosis information from different angles.Patient-level prognostic prediction based on MIL strategies still require further research.This includes how to effectively characterize the instance-level information related to patient prognosis in MRI or WSI,how to determine the importance of the instances for the prognostic labels,and how to enhance the interpretability of the models.This dissertation focused on the prognosis prediction of patients with rectal cancer.The author first proposed a MIL method based on attention-guided deep feature fusion using MRI images,which initially achieved the aggregation of instance-level information to patient-level information.Then,a MIL method based on tumor mutation burden(TMB)-guided deep feature fusion was proposed using WSI images,which expanded the interpretability of MIL method.Finally,a domain-adaptive method based on federated learning framework was proposed to enable the proposed multi-instance prognosis prediction method to adapt to future large-scale clinical validation and clinical promotion,while ensuring the accuracy of model prediction while achieving patient privacy protection.The main work and contributions of this dissertation are as follows.First,a MIL method for attention-guided MRI deep feature fusion was proposed.Deep learning features for rectal cancer multi-parameter MRI images were first encoded separately by independent pre-trained convolutional neural networks(CNNs).Subsequently,patientlevel prognosis prediction was realized by multi-parameter MRI deep learning feature fusion under the attention mechanism.For the prediction of patients’ three-year distant metastasisfree survival(DMFS)after surgery,pre-trained 2D CNNs with transfer learning was used to train binary DM risk classification models.Based on the models’ outputs,instance-level features of DM risk specificity by MRI with different parameters were derived.After achieving instance-level feature fusion through summation operations,a multi-parameter MRI prognostic marker independent of clinicopathological factors was developed based on Cox regression.A joint model was constructed by combining this marker with clinicopathological factors.Through the joint model,the risk of DM in patients with different responses to neoadjuvant therapy can be effectively assessed.For the prediction of five-year overall survival(OS)of patients after surgery,3D CNN combined with a gated attention mechanism was used to achieve global feature encoding and patient-level information fusion of multi-parameter MRIs.This strategy showed the best prediction performance compared with other feature extraction and feature fusion methods in five-fold cross-validation,with an average prediction C-index of 0.668.Secondly,a MIL method for TMB-guided WSI deep feature fusion is proposed.Compared with non-invasive MRI,WSI contains gigapixels,in which information expressed by molecular markers may be buried.Therefore,this dissertation used prognosis-related TMB as an intermediate variable to achieve information aggregation between WSI instances by constructing a TMB classification model and performing correlation analysis.On this basis,WSI prognostic markers related to TMB were developed.The feasibility of the above twostage construction method of prognostic markers was first validated on the existing dataset.In the preliminary validation,based on manually annotated regions of interest(ROI),WSIlevel pre-trained deep learning features were aggregated through average pooling,and the threshold for high-and low-TMB were determined through segmented linear regression analysis.The TMB prediction model was constructed through feature selection and L2 regularized logistic regression.The area under the curve(AUC)for the TMB prediction of this model on the internal validation set was 0.813.The prognostic markers obtained according to the TMB prediction model achieve significant stratification of OS risk of patients,especially advanced patient groups,on the TCGA dataset.Subsequently,this dissertation further proposed a multi-instance learning method based on a dual attention mechanism without ROI annotation.The superiority of the method was demonstrated by the prediction result of the TMB prediction model on the TCGA Rectal Adenocarcinoma public dataset,where the AUC for the prediction model was 0.818.The method achieved a 7%improvement in prediction accuracy compared to the global feature average pooling scheme without ROI annotation.In addition,significant stratification of patients’ risk of OS could be achieved based on the model output.Finally,a federated domain adaptation prognostic prediction method based on Amplitude Dual Attention(ADA)was presented.The method was based on the federated learning framework,which mitigated the performance degradation of distributed learning due to nonindependent identically distribution(non-IID)of multicenter data through a federated domain adaptation module called ADA.The goal was to achieve further generalization and possible large-scale clinical validation of the proposed multi-instance prognostic prediction method under patient privacy protection.This method first transformed the MRI into the frequency domain space by Fourier transform.Then,according to the local amplitude spectral attention module and global amplitude spectral attention module,the patient-level low-frequency information and the inter-center low-frequency information were constrained to the same space,respectively.This enabled style migration and unification of data between different centers.In downstream task applications,the OS prediction model constructed based on this method achieved stable performance improvement under multiple federated learning base frameworks.The average C-index for five-fold cross-validation was 0.657,which was closest to the model prediction accuracy under centralized learning.Further comparative experimental results showed that the model prediction accuracy based on the proposed method in this study outperformed the existing unsupervised domain adaptation method.In particular,the model trained under the ADA+Fed Prox strategy achieved equal or even better prediction accuracy than the model trained under the centralized training strategy in the validation set of each independent center.Significant stratification of patients at high and low risk of OS across centers could be achieved through the model output,demonstrating its potential as a possible clinical aid for decision-making tools.In summary,this dissertation provided methodological innovations based on traditional MIL for the characteristics of tumor image data.Application validation and in-depth discussion were carried out in the prognosis prediction of rectal cancer.Specifically,it included the development and validation of MRI-and WSI-based MIL methods,and provided an effective solution for further clinical validation and generalization of the MIL methods under patient privacy protection through the federated learning framework.