| [Background and Objective]The bone tumors are relatively rare,but the malignant bone tumors are the main cause of cancer-related death in adolescent patients.Accurate classification of bone tumors in early stage and appropriate treatment strategies are important to improve the survival rate of patients.Therefore,the preoperative classification of bone tumors plays a significant role in clinical practice.Digital radiography has become the preferred means for the diagnosis of bone tumors because of its high spatial resolution,convenience and lower price.In clinical practice,the classification of bone tumors mainly depends on the comprehensive analysis of the lesions by radiologists.They make the final diagnosis base on the extracted imaging features and the clinical information.However,it is difficult for junior radiologists to identify important imaging features which can provide reliable classification of bone tumor for treatment owing to the complex imaging features of bone tumors.The study had constructed four machine learning classification models of bone tumors based on the extracted imaging features and clinical information,which could figure out the most important image features for the classification of bone tumors.In this study,the classification performance and clinical value of each model were evaluated according to the histological results.The study was divided into two parts.The first part constructed four models,which were logistic regression(LR)model,decision tree(DT)model,random forest(RF)model and support vector machine(SVM)model.The SHAP Value algorithm was used to evaluate the importance of each feature in the diagnosis of bone tumors during the model training.And the area under curve(AUC)value was used to evaluate the performance of the model in the test set.In the second part,three radiologists with different bone tumor diagnostic experience were chose for the reader study with or without the assistance of the four models for the purpose of figure out the practice value in the clinical work.The reading results of the three radiologists and the selected optimal model were compared.[Materials and Methods]1.Clinical informationThe clinical data of 557 patients with bone tumors confirmed by pathology in Nanfang Hospital of Southern Medical University from January 2014 to September 2019 were collected retrospectively,including 289 cases of benign,118 cases of intermediate and 150 cases of malignant.In addition,239 cases of bone tumors were collected from Foshan Hospital of TCM and University of Hong Kong-Shenzhen Hospital from May 2018 to August 2019,including 123 cases of benign,51 cases of intermediate type and 65 cases of malignant.This study included patients with bone tumor who 1)underwent at least one preoperative conventional radiographic examination in one of the three academic medical centers and 2)had a single lesion and 3)had a pathological diagnosis of a benign or malignant bone tumor via biopsy Exclusion criteria were:1)the pathological results were unclear or 2)there was a foreign body in the conventional radiographic images or 3)the recurrent lesions after operation or 4)the evaluation of the lesion was limited due to the overlap of tissues2.Feature extractionPreoperative conventional radiographic features and potentially relevant clinical data were extracted and compiled in a structured database by three musculoskeletal system radiologists(with over 5 years of working experience in bone tumor diagnosis)who were blinded to the patients’pathological diagnoses.The radiologists independently scored each conventional radiographic feature and the clinical data,and the scores were averaged across radiologists.Clinical information was obtained through medical records.3.Machine learning modelMachine learning models were constructed based on the imaging features and clinical information extracted by radiologists.The binary models were built to classify bone tumors as benign or malignant.And the tertiary models were built to classify tumors as benign,malignant or intermediate.Shapley additive explanations was used to understand the most important conventional radiographic features for the classification of bone tumors.The study compared the difference between the four machine learning models with AUC values.4.Reader studyThree radiologists with low seniority(with 1-3 years working experience),middle-seniority(with 3-5 years working experience)and high seniority(with over 5 years working experience)were selected to complete the binary and tertiary classification of bone tumors independently.The reader study divided into three mode.Each radiologist should complete reading experiments and compare their diagnostic performance.At the same time,the differences between the radiologists and the optimal model were analyzed.In the first mode,each physician scored the lesion of bone tumor after completing reader study independently.In second-reading mode,each radiologist completed the reader study and scored the bone tumor with the assistance of the important features.In third-reading mode,each radiologist completed the reader study and scored the bone tumor with the assistance of the important features and output result.The score was selected on a scale of 10.A score of 0 means definitely benign,and a score of 10 means definitely malignant.In the binary classification,1-2 points:almost certainly benign;3-4 points:most likely benign;5 points:may be benign;6 points:may be malignant;7-8 points:most likely malignant;9-10 points:almost certainly malignant.In the tertiary classifications,1-2 points:most likely benign;3-4 points:may be benign;5-6 points:may be intermediate;7 points:most likely intermediate;8-9 points:may be malignant;10 points:very likely malignant.5.Statistical analysisStatistical analysis was conducted using SPSS version 20.0 and MedCalc software.Measurement data were expressed by mean ± standard deviation.One-way analysis of variance(ANOVA)was used to compare clinical variables in patients with benign,malignant or intermediate bone tumors.LSD method was selected when the variance was homogeneous,and Welch approximate analysis of variance was selected when the variance was uneven.The differences of AUC values between machine learning models and radiologists in the diagnosis of bone tumors were compared by using the DeLong test.P<0.05 was considered to indicate a statistically significant difference.[Results]The first part:1.Data sets of binary and tertiary classification machine learning modelsIn the binary classification models,the training set was composed of 438 patients,including 298 patients of benign bone tumors and 140 patients of malignant bone tumors.The test set was composed of 189 patients,including 114 patients of benign bone tumors and 75 patients of malignant bone tumors.In the tertiary classification model,the training set was composed of 557 patients,including 289 patients of benign bone tumors,118 patients of intermediate bone tumors and 150 patients of malignant bone tumors.The test set was composed of 239 patients,including 123 patients of benign bone tumors,51 patients of intermediate bone tumors and 65 patients of malignant bone tumors2.Clinical information of bone tumor patientsHistological type of bone tumors was significantly associated with all involved clinical parameters except gender(P>0.05).Patients with a malignant bone tumor were significantly older than those with a benign bone tumor and intermediate bone tumor(mean age,33 vs.23 vs.24 years;P<0.001).3.Feature importance evaluationAccording to the SHAP value algorithm,the most important feature for the classification of bone tumors in each model was the margin of the lesion.In the binary classification,each model evaluated the lesion margin,cortex involvement and high density components as more important features.But in the tertiary classification,lesion margin and high density components were the two most important features in all the four models4.The performance of machine learning modelIn the binary classification,the AUC values of DT,RF,LR,and SVM models were 0.917,0.973,0.973,and 0.976.The AUC value of RF and SVM models were higher than that of DT model(P<0.05).There was no significant difference in AUC values among LR,RF and SVM models(P>0.05).In the tertiary classification,the average AUC values of DT,RF,LR and SVM models were 0.821,0.935,0.930 and 0.944.The AUC values of LR,RF and SVM models were higher than that of DT model(P<0.05).There was no significant difference in AUC values between RF and SVM models,LR and SVM models(P>0.05).However,there was significant difference in AUC value between LR model and RF model in the diagnosis of benign and malignant bone tumors(P<0.05).Combined with the results of binary and tertiary classification,the SVM model was selected as the best performance model in this study.The second part:1.Classification performance and comparative results of bone tumors by different seniority radiologistsThe radiologists completed the reader study without any assistant information first.In the binary classification,the AUC values of low,middle and senior radiologists were 0.894,0.908 and 0.991,respectively.In the tertiary categories,the AUC values of benign bone tumors diagnosed by three radiologists were 0.821,0.825 and 0.949,respectively.For malignant bone tumors,the values were 0.842,0.886 and 0.988,respectively.For intermediate bone tumors,the values were 0.763,0.732 and 0.737,respectively.The AUC value of senior radiologist was higher than that of junior and middle-seniority radiologists(P<0.05).There was no significant difference between junior and middle-seniority radiologists(P>0.05).The classification performance of SVM model was comparable to senior radiologists(P>0.05)and higher than that of low and middle-seniority radiologists2.Diagnostic performance of radiologists assisted by machine learning model①The same radiologists completed the second-reading mode with the assistant information of important features.The diagnostic AUC values of low,middle and senior radiologists were higher than that of the first reading task,but the difference was not statistically significant(P>0.05).With the assistance of important features,there was no significant difference in the diagnostic performance of bone tumor classification between SVM model and junior,middle-seniority radiologists(P>0.05).In the tertiary classifications of bone tumors,the diagnostic performance of senior radiologist was better than that of SVM model(P<0.05).②The radiologists completed the third-reading mode with the assistant information of the important features and the output results of the model.In the binary classification,the AUC values of low,middle and senior radiologists for the classification of bone tumors were higher than those of the first-reading mode.There was no significant difference among the three reading modes and between middle-seniority doctors and SVM model(P>0.05).In the tertiary classification,the AUC value of junior radiologist was higher than that of the first-reading model,and the difference was statistically significant(P<0.05).The AUC values of benign and malignant bone tumors diagnosed by junior radiologist were lower than that of SVM model,and the difference was statistically significant(P<0.05)。The AUC value of the third-reading mode of middle-seniority radiologist was higher than that of the previous two reading modes,and the difference was statistically significant(P<0.05).In the tertiary classification,there was no significant difference between middle-seniority radiologist and SVM model(P>0.05).The AUC value of the third-reading mode of senior radiologist was higher than that of the first-reading mode.And there was no significant difference among the three reading modes(P>0.05).In addition,the AUC value of senior radiologist was higher than that of SVM model,and the difference was statistically significant(P<0.05).[Conclusion]1.At present,many machine learning models are similar to the "black box"structure,which can only provide the results without explaining its principle.The study constructed four bone tumor classification models based on the extracted digital radiography features and clinical information.The SHAP value was used to identify the most important features for the classification of bone tumors and to explain the results of the model.In the binary and tertiary classification tasks,the most important feature obtained by each model was the margin of lesion.The result suggested that radiologists should pay attention to the value of the margin in the differential diagnosis of bone tumors2.The accuracy of the four machine learning models in the diagnosis of bone tumors was different.For benign bone tumors,the osteochondroma,non-ossifying fibroma and endophytic chondroma got higher accuracy,and the hemangioma and benign fibrous histiocytoma got lower accuracy.For malignant bone tumors,the osteosarcoma,metastatic tumor and bone lymphoma got higher accuracy,on the contrary,the malignant fibrous histiocytoma and Ewing’s sarcoma was lower3.All the machine learning models constructed in this study provide reliable classification results(AUC>0.80),which could provide assistant information for the clinical practice.Generally speaking,the classification performance of RF model,LR model and SVM model were better than that of DT model.The AUC value of SVM model was the highest,and the classification performance was comparable to the senior radiologists.4.The AUC values of each classification models in the tertiary classification tasks were lower than that of the binary classification.Intermediate bone tumors got the lowest diagnostic performance whether in the three radiologists or the four machine learning models.This may be due to the blurred boundaries of biological behavior and the lacking of important differential imaging features of the intermediate bone tumors,which resulted in the misdiagnosis more easily both for the radiologists and machine learning models.5.In the reader study,there were difference in the classification performance of bone tumors between different radiologists.Due to the lacking experience in the diagnosis of bone tumors,the performance of low-seniority and middle-seniority radiologists were lower than that of senior radiologist.All three radiologists showed improved diagnostic performance of bone tumors with the assistance of SVM model,especially in the case of providing both output result of the model and the important radiography features. |
PDF Full Text Request