Research And Development Of Thyroid Nodule Automatic Recognition And Diagnosis System Based On Deep Learning In Thyroid Ultrasound Dynamic Video

BackgroundThyroid nodules are a common clinical problem.The prevalence of thyroid nodules detected by ultrasound is 19-68%worldwide.Most thyroid nodules are benign,and only 7-15%of thyroid nodules are malignant.Ultrasonography is the imaging method of choice for evaluating thyroid diseases.Ultrasonography can judge benign and malignant nodules according to their image characteristics,such as hypoechoic nodules,taller-than-wide,lobulated or irregular edges,microcalcification and extrathyroidal extension,which are considered to be related to malignancy.There are many guidelines in China and abroad to stratify the risk(RSS)of malignancy of thyroid nodules according to their ultrasound characteristics,and make management recommendations on whether to perform fine needle aspiration biopsy(FNA)based on their size.Some of these systems are called Thyroid Imaging and Reporting Data System(TI-RADS).The diversity,complexity,and lack of consistency of these systems lead to inconsistent risk predictions for diagnosis.At the same time,with the increase of thyroid patients,the risk classification of each nodule increases the workload,which may affect the diagnostic accuracy of already busy sonographers,leading to inevitable diagnostic bias and unnecessary FNA.In order to overcome the limitations of ultrasound diagnosis,computer aided diagnosis(CAD)systems based on Artificial intelligence(AI),especially deep learning methods,have been introduced for the diagnosis of thyroid nodules.Many studies have reported the potential role of these systems in thyroid cancer diagnosis and have shown comparable or even superior diagnostic performance to experienced sonographers.Our research is to develop an intelligent thyroid ultrasound scanning system based on deep learning,which can identify,detect and diagnose thyroid nodules from thyroid ultrasound dynamic videos,and detect and segment the tissues around the thyroid,including anterior jugular muscle,carotid artery,internal jugular vein,trachea,esophagus and cricoid cartilage,etc.This study aims to develop a thyroid ultrasound examination system that fully conforms to the examination process and diagnostic thinking of sonographers,so as to truly reduce the workload of sonographers,improve the detection rate and diagnostic accuracy of thyroid nodules,and embed artificial intelligence into the whole process of thyroid ultrasound examination.Our method is the first to propose a method to detect and track thyroid nodules and surrounding tissues in thyroid ultrasound video,and to classify the nodules.Part one Deep learning-based ultrasonic dynamic video detection and segmentation of thyroid gland and its surrounding cervical soft tissuesObjective:To establish and verify a deep learning method(Cascade region-based convolutional neural network,R-CNN)based on ultrasound videos for automatic detection and segmentation of the thyroid gland and its surrounding tissues,so as to reduce the workload of sonographer and improve the detection and diagnosis rate of thyroid diseases.Methods:71 normal thyroid ultrasound videos were collected,including five standard videos(left and right lobe transverse scan,isthmus transverse scan,left and right lobe longitudinal scan)for each patient,and 355 thyroid ultrasound videos were finally obtained.59 patients were the training dataset,12 patients were the validation dataset,and 9 patients were the test dataset.The sonographer labeled the neck tissues,including anterior cervical muscle,cricoid cartilage,trachea,thyroid gland,endothyroid vessels,carotid artery,internal jugular vein,and esophagus.A large dataset was constructed to train and test the deep learning method.The performance was evaluated using the COCO metrics AP,AP50,and AP75.We compared the Cascade R-CNN with a state-of-the-art method CenterMask in the test dataset.Results:We annotated 166817,34364 and 29227 regions in training,validation and testing samples.The model could achieve a good detection performance for the thyroid left lobe,right lobe,isthmus,muscles,trachea,carotid artery,and jugular vein;the AP50 of these tissues were 0.865,0.875,0.891,0.961,0.966,0.97,and 0.918,respectively.In addition,the model showed good segmentation performance for the muscles,trachea,and carotid artery;the AP50 of these tissues were 0.96,0.966,and 0.978,respectively.For the left lobe,right lobe,isthmus,esophagus,and jugular vein,AP50 was≥0.86.However,the segmentation results for the cricoid cartilage and endothyroid vessels were not high(AP50 of 0.539 and 0.485,respectively).For fair comparison,the performance of Cascade R-CNN is better than that of CenterMask for detection and segmentation tasks.The difference was statistically significant(P<0.05).Conclusions:The new method could successfully detect and segment the thyroid gland and its surrounding tissues.Part Two Deep learning-based ultrasonic dynamic video detection and tracking of thyroid nodules and surrounding tissuesObjective:To establish and verify a deep learning method to detect and track thyroid nodules and surrounding tissues in ultrasound dynamic videos,aiming to accurately detect and track thyroid nodules in ultrasound videos.Methods:1734 thyroid dynamic ultrasound videos were collected from 769 patients in our hospital,including 175 males and 594 females,with an average age of 45.98±13.08 years.The training dataset consisted of 940 videos from 419 patients,including 1192 thyroid nodules.The test dataset consisted of 794 videos from 350 patients,including 857 thyroid nodules.We proposed a deep learning-based detection and tracking framework to detect thyroid nodule,thyroid gland,anterior cervical muscle,trachea,esophagus,carotid artery,and internal jugular vein in thyroid ultrasound videos.The COCO metrics(mAP,mAP50,AP50)were used to evaluate the detection performance.We compared the detection performance of the proposed method with the state-of-the-art methods in the test dataset.At the same time,we collected 66 patients from other hospital as an external test dataset to verify our model,including 13 males and 53 females,with an average age of 50.18±11.68.A total of 132 nodules were obtained from 144 videos collected.Results:1.The mAP50 of our method for the detection of thyroid nodules and their surrounding tissues is 0.880.The AP50 of Sort、DeepSort、Fast R-CNN,RetinaNet,Yolov3,Yolov4 and Yolov5 were 0.851,0.604,0.833,0.807,0.733,0.841 and 0.884,respectively.The detection performance of our method is comparable to the latest Yolov model Yolov5,and better than other detection methods.2 Our method for thyroid nodule results show AP50 is 0.657,and the AP50 of Sort,DeepSort,Fast R-CNN,RetinaNet,Yolov3,Yolov4 and Yolov5 were 0.594,0.068,0.645,0.582,0.478,0.626 and 0.677,respectively.The detection performance of our method for nodules is slightly lower than that of Yolov5,but it is better than other detection methods.3.The mAP50 of external datatest set for thyroid nodules and surrounding tissues was 0.850,and the AP50 of nodule was 0.510,which was not as good as that of internal test dataset.There was no significant difference between internal test set and external test set(P>0.05).Conclusion:Our proposed detection and tracking framework can not only detect thyroid nodules,but also track and monitor the surrounding tissues.Part Three Automatic thyroid nodules diagnosis in Ultrasound videos based on deep learningObjective:The deep learning method is applied to develop an automatic thyroid nodule diagnosis system,which can realize the classification of each nodule in the video,provide diagnostic opinions for sonographer,and reduce the workload of sonographer.Methods:1734 thyroid dynamic ultrasound scanning videos were collected from 769 patients in our hospital,with a total of 2049 nodules.The training dataset included 1192 thyroid nodules,including 748 benign nodules and 444 malignant nodules.The test dataset included 857 thyroid nodules,of which 403 were benign and 454 were malignant.We propose a deep learning-based framework that uses the 2017 ACR TI-RADS Thyroid Ultrasound Reporting Dictionary to interpret the features of each image,obtain the corresponding frame-level TI-RADS score and eigenvector,and build a model for the temporal context information in the eigenvector sequence.Pathological diagnosis is the gold standard for final diagnosis.The diagnostic performance of the proposed method was compared with other state-of-the-art methods for thyroid nodules.In order to analyze the classification performance of the system in more detail,we compared the classification results of nodules with different sizes.At the same time,132 nodules were classified from 144 videos in the external test dataset to further test and validate our model.Results:1.The diagnostic accuracy(0.7480),precision(0.8287),recall(0.6608),F1 score(0.7353)and area AUC under the receiver operating characteristic(ROC)curve(0.8154)of our method for malignant thyroid nodules exceeded those of other state-of-the-art classification methods(TRN,Slowfast,TDN,C3D,I3D,P3D,ConvLSTM,LRCNs,ARTNet,Res3D,Res21D).The specificity of our method is 0.8462,which is lower than some other methods.2.We divided the size of thyroid nodules into three categories:<10mm,10-20mm and>20mm.The results showed that the F1 scores of the proposed method for the classification of nodules of each size were 0.7703,0.7917 and 0.9474,respectively.The larger the nodule,the better the classification effect.3.The accuracy,precision,recall,F1 score,specificity and AUC of the external test set for the diagnosis of malignant thyroid nodules were 0.5985,0.5469,0.5963,0.5691,0.6027 and 0.6772,respectively.The diagnostic performance of the external test set was not as good as that of the internal test set.Conclusions:In this study,we proposed a deep learning-based thyroid ultrasound video computer-aided diagnosis framework for thyroid nodule classification.The results show that the proposed framework is superior to other tate-of-the-art classification methods.This framework can not only help sonographers to diagnose,but also strive to achieve the goal of computer-aided diagnosis to reduce the workload of sonographers.