Prediction Of Treatment Response To Neoadjuvant Therapy Based On Longitudinal Ultrasound Images Of Breast Cancer Patients By Deep Learning Techniques

BackgroundBreast cancer is the most common cancer in the world.Neoadjuvant therapy(NAT)is the standard of care for patients with breast cancer.Potential benefits of NAT include tumour downsizing,increasing the probability of breast-conserving and limited axillary surgery.However,breast cancer is heterogeneous with variable responses to NAT,with pathological complete response(pCR)rates ranging from 20%to 80%.Accurate prediction of pCR allows for early intervention for non-pCR patients to increase pCR rates and guides clinicians in choosing more limited surgery in the breast and axilla.However,no reliable biomarkers currently exist to aid in pCR prediction.Previous studies have evaluated tumour response in breast cancer based on ultrasound images using the deep learning method.However,in traditional quantitative image analysis,tumour segmentation is delineated manually by sonographer,which is time-consuming and has inter/intra-observer variability.Moreover,previous studies did not consider the therapy-induced changes and effectively extract dynamic information from longitudinal images.Therefore,it is urgent to investigate whether the deep learning method could enable the automated segmentation of breast cancer ultrasound and mining of the dynamic change information from serial ultrasound images during NAC for breast cancer,thus achieving qualification of the temporal heterogeneity and accurate PCR prediction.PurposeIn response to the clinical challenge that the response to NAT in breast cancer is difficult to be predicted,this study aimed to 1)train and validate a segmentation model based on the UNet to achieve automatic and accurate segmentation of breast cancer ultrasound during NAC.2)based on the automatic segmented breast ultrasound in part one,we further developed and validated a serial ultrasonography assessment system(SUAS)for the prediction of treatment response for breast cancer patients using serial ultrasound images before,during the first/second cycle of,and after NAT.3)develop and validate Siamese multi-task network(SMTN)that combined tumour segmentation and pCR prediction using dynamic change information from longitudinal ultrasound images before and after the first/second cycle of NAT for HER2-positive breast cancer to predict early treatment response.Methods1)A total of 801 patients with biopsy-proven breast cancer were retrospectively enrolled from three institutions between May 2015 to June 2020 and were split into a training cohort(439 patients)and two external test cohorts(212 and 150 patients).The region of interest(ROI)was manually delineated on the ultrasound images of the largest cross-section of breast cancer during NAT using ITK-SNAP.We trained and validated the automatic segmentation model based on the UNet and the performance of the segmentation model was evaluated using the Dice similarity coefficient(DCIE).2)Study population was the same as in part 1.The training cohort was further chronologically divided into a training cohort(242 patients)and an internal validation cohort(197 patients).Three imaging signatures were constructed from the serial ultrasonographic features before(pretreatment signature),during the first or second cycle of(early-stage treatment signature),and after(posttreatment signature)NAT based on auto-segmentation by U-net.The SUAS was constructed by subsequently integrating the pre,early-stage,and posttreatment signatures,and the incremental performance was analysed.3)A total of 393 patients with biopsy-proven HER2-positive breast cancer were retrospectively enrolled from three hospitals in china between December 16,2013,and March 05,2021,and allocated into a training cohort(215 patients)and two external validation cohorts(95 patients and 83 patients).The proposed SMTN consists of two subnetworks that could be joined at multiple layers,which allowed for the integration of multi-scale features and extraction of dynamic information from longitudinal ultrasound images before and after the first/second cycles of NAT.Results1)The UNet achieved satisfactory segmentation performance in two external test cohorts,with DICEs of 0.806 and 0.785 respectively.Moreover,our model demonstrated good segmentation accuracy for the images throughout the three phases(DICE>0.750).Meanwhile,our results showed that the automated model could perform effective segmentation not only for large-size(equal to or more than 2 centimeters)but also for small-sized lesions(less than 2 centimeters,DICE>0.725)2)The SUAS yielded a favour performance in predicting pCR,with areas under the receiver operating characteristic curve(AUCs)of 0.927(95%confidence interval[CI]:0.891-0.963)and 0.914(95%CI:0,853-0.976),compared with those of the clinicopathological prediction model(0.734(95%CI:0.665-0.804)and 0.610(95%CI:0.504-0.716)),and radiologist interpretation(0.632(95%CI:0.570-0.693)and 0.724(95%CI:0.644-0.804))in the external test cohorts.Furthermore,similar results were also observed in the early-stage treatment of NAT(AUC:0.874(95%CI:0.793-0.955)to 0.897(95%CI:0.851-0.943))in the external test cohorts.3)The SMTN yielded AUC values of 0.986(95%CI:0.977-0.995),0.902(95%CI:0.856-0.948),and 0.957(95%CI:0.924-0.990)in the training cohort and two external validation cohorts,respectively,which were significantly higher than those of the clinical model(AUC:0.524-0.588).Moreover,272 of 279(97.5%)nonpCR patients and 82.5%(94/114)pCR patients were successfully identified by the SMTN,suggesting that they could be benefit from regime adjustment at the early stage and limited breast and axilla surgery.ConclusionWe demonstrate that deep learning-based models can segment breast cancer ultrasound images accurately and integrating serial ultrasound features throughout NAT can predict pCR with favour performances,which can facilitate individualized treatment strategies.