The Evaluation Of A Assisted Diagnosis System Based On Deep Convolutional Neural Network Model In Diagnosis Early Gastric Cancer

Objective: An Artificial intelligence(AI)system based on deep convolutional neural network(DCNN)was developed to assist the diagnosis of early gastric cancer.It is expected that the system will improve the ability of endoscopists to diagnose early gastric cancer.Materials and Methods: This study was divided into three parts.Firstly,multi-center white light endoscopic images of early gastric cancer were retrospectively collected.After inclusion and exclusion criteria,a total of 3400 white light endoscopic images of early gastric cancer and 8600 images of non-cancer were used as the training set.Four endoscopists classified and labeled these images according to pathological results and staining images.Second,the training dataset images denoising,transformation,cropping and other preprocessing,Center Net methods was used for model training,and then it was developed into a software system in order to convenient used for clinicians to apply.Third,the part of application in systematic clinical,the clinical verification aspects of the system was divided into the following aspects:(1)The diagnostic effectiveness of the DCNN model in the internal verification dataset and the external verification dataset;(2)Comparison of diagnostic efficacy between DCNN model and endoscopists who came from county hospital,city hospital and provincial hospital;(3)Comparison of diagnostic efficacy of endoscopists before and after the assisted of DCNN model;(4)Comparison of diagnostic efficacy of endoscopists and DCNN model in detecting and locating lesions;(4)Comparison of diagnostic efficacy of endoscopists and DCNN model in diagnosis benign lesions.The validation dataset came from multi-center,and independent from training dataset.There was 12 endoscopists came from endoscopy centers of county hospital,city hospital and provincial hospital,and they were dividedinto senior group and junior group.In terms of all verification datasets,all images and videos including early gastric cancer and benign lesions were randomly arranged before each diagnosis,and all of endoscopists were not participated in the collection and labeling of images.During the verification process,the DCNN model and endoscopists were asked to independently diagnosis the images,record the results and diagnosis time.Accuracy、sensitivity、specificity、positive predictive value、(PPV)、negative predictive value(NPV)、Area under curve(AUC)、 consistent Cohen’s Kappa index and diagnostic time were used to evaluate the diagnostic efficacy.The causes of false positive and false negative by DCNN model and endoscopists were analyzed.All data were processed and plotted using SPSS26(SPSS Inc.,Chicago,IL,USA)and Graph Pad 9.4.1(Graph Pad Software Inc.,San Diego,CA,USA).Results: In the first stage,43,120 images were collected from 1,475 patients which was come from three centers.After the inclusion and exclusion criteria(criteria 1 and2),total 31,120 images were excluded,3400 images were included for early gastric cancer and 8600 images were included for benign lesion,four endoscopists was asked to label the lesion range.In order to ensure the accuracy of the results,crossvalidation was implemented.The second stage: We preprocessed the training images,including denoising,transformation and clipping,and then we compared four target detection technologies : Center Net,Faster R-CNN,YOLOv3 and Retina Net.We found that the Center Net performed best in terms of accuracy and speed,and then we investigate the use habits of endoscopists,according to the outcomes we developed the DL model as a software system in order for convenient used.The third stage,the clinical application of DCNN model.(1)Evaluation of diagnostic effectiveness of DCNN model: In the image testing set,the diagnostic AUC of the model in the internal and external image test sets was 0.917 and 0.925,respectively.The sensitivity was 93.37%(95% Confidence interval(CI),91.09%-95.12%)and 95.0%(95% CI,91.38%-97.20%)respectively,and the specificity was 90.07%(95% CI,87.59%-92.10%)in test1 and90.06%(95% CI,86.14%-92.99%)in test2;In the video testing set,the diagnostic AUC of the internal and external image testing sets were 0.931 and 0.917,respectively.the diagnostic sensitivity were 96.92%(95%CI,88.83%-99.78%)and92.86%(95%CI,80.30%-98.23%),respectively,and the specificity was 89.23%(95%CI,79.11%-94.98%)in test1 and 90.48%(95%CI,77.38%-96.79%)in test2;(2)Comparison of diagnostic efficacy between the DCNN model and endoscopists: In the image test set,the sensitivity and NPV of the DCNN model were higher than those of all endoscopists;the accuracy was lower than that of two endoscopists(91.62%vs.93.10%-93.25%),and the specificity was lower than those of three endoscopists(90.07% vs.93.43%-94.31%).The sensitivity of provincial endoscopists was higher than that of municipal endoscopists,and the municipal endoscopists which was higher than that of county endoscopists(80.79%-82.12% > 78.15%-79.47% >68.21%-70.86%),and the endoscopists’ accuracy 、 sensitivity in county hospitals,municipal hospitals and provincial hospitals showed a gradually increasing trend from low seniority to high seniority.We further analyzed the causes of false positives and false negatives produced by DCNN model and endoscopists,the top three reasons for false positives were gastritis,atrophy or redness(42.65%)、mucus(10.29%)and folding(8.82%).Meanwhile,the cause of the endoscopist’s false positives were atrophy or redness(57.04%)、 benign ulcer(11.39%)、 and proliferative adenoma(10.04%).The top three reasons for false negative in DCNN model were different shooting angles(40.00%)、 lesions less than 10 mm in diameter(25.00%)and shooting distance(15.00%),however,endoscopists are prone to cause the following three ways of false negatives:inflammation-associated mucosa(32.57%)、 lesions less than 10 mm in diameter(19.94%),and benign ulcers(16.25%).In the video testing set,the accuracy、 sensitivity and NPV of the DCNN model were higher than those of all endoscopists,the sensitivity of endoscopists in the junior group was generally lower than that in the senior group.In the junior group,the sensitivity of endoscopists in the county group was lower than that in the municipal and provincial groups(73.85%-75.38%vs.76.92%-80%),the sensitivity of the municipal group was higher than that of the provincial group(78.46%-80% vs.76.92%-80%).The sensitivity of endoscopists in the senior group increased gradually in from the county hospital endoscopists,city hospital endoscopists to provincial level.(3)Comparison of efficacy of endoscopists before and after the assistance of DCNN model: In the image testing set,the accuracy of endoscopists after the assistance of DCNN was significantly improved(93.25%-96.66% vs.76.96%-93.25%).After assistance,the accuracy of county endoscopists、city endoscopists and provincial endoscopists was comparable(94.96%-95.11% vs.95.42%-96.66% vs.93.25%-96.51%).After assistance, the diagnostic accuracy of endoscopists in county hospitals increased by 9.46%(95%CI,7.98%-11.19%)to18.15%(95% CI,16.14%-20.35%)、the diagnostic accuracy of endoscopists in municipal hospital increased by 5.90%(95% CI,4.73%-7.32%)to14.82%(95% CI,12.98%-16.86%)、 the diagnostic accuracy of endoscopists in county provincial hospital increased by 0.62%(95% CI,0.29%-1.24%)to 11.17%(95% CI,9.56%-13.01%).The diagnostic time of a single image in the DCNN model was 0.028 s,and the diagnostic time of endoscopists before and after the assistance was significantly shortened(7.09±0.65 s vs 5.05±0.55 s,P < 0.001).The range of Kappa values of endoscopists increased from 0.705-0.753 to 0.768-0.890.In the video testing set,the diagnostic accuracy of endoscopists after assistance was significantly improved(74.62%-90.77% vs.80.08%-90.77%),however there was a statistical difference between the junior and senior groups(86.41% vs.91.03%,P=0.03).After assistance,the accuracy of the county endoscopists increased by 5.38%(95% CI,2.44%-10.89%)to 9.23%(95% CI,5.23%-15.57%)、the accuracy of the municipal endoscopists increased by 4.62%(95%CI,1.92-9.92%)to 7.69%(95%CI,4.08%-13.73%),and the accuracy of the provincial endoscopists increased by 0-9.23%(95%CI,5.23%-15.57%).The diagnostic time of endoscopists before and after assistance was shortened(2392.17±7.77 s vs.2378.34±23.51 s,P < 0.02),and the range of Kappa values of endoscopists was increased from 0.657-0.793 to 0.738-0.905.(4)Comparison of DCNN model and endoscopists in lesion detection and localization:In the internal test set,the AUC of the DCNN model、the junior endoscopists and the senior endoscopists were 0.886、0.771 and 0.889,respectively.The accuracy of the DCNN model was 10.47% higher than that of the junior endoscopists(95%CI,8.91-12.27%,P<0.05),and was 1.16% lower than that of senior endoscopists(95%CI,0.69%-1.93%,P > 0.05).The sensitivity of DCNN model was higher than that of the junior endoscopists and lower than that of the senior endoscopists.In the external test,The AUC of the DCNN model、the junior endoscopists and the senior endoscopists were 0.921、0.767 and 0.901 respectively,the accuracy of the DCNN model was14.67% higher than that of the junior endoscopists(95%CI,11.84%-17.81%,P <0.01),and was 1.94% higher than senior endoscopists(95%CI,1.25%-2.96%,P > 0.05).the sensitivity of DCNN model was higher than that of all endoscopists.(5)Diagnostic efficacy of benign lesions: The diagnostic sensitivity of DCNN model for gastric benign ulcer was 95.2%(95%CI,88.84%-95.46%),higher than that of all endoscopists.The diagnostic sensitivity of gastric polyps/gastric adenomas was 92%(95%CI,87.91%-94.82%),comparable to that of senior endoscopists.The sensitivity of DCNN model for the diagnosis of atrophic gastritis/normal gastric mucosa was88%(95%CI,83.35%-91.50%),higher than that of all endoscopists.However,the PPV was lower than that of endoscopists in the junior group(92.44% vs.95.24%,P <0.05)and senior endoscopists group(92.44% vs.95.5%,P < 0.05).Conclusions: Firstly,We developed a diagnostic system for early gastric cancer based on DCNN model,and its diagnostic efficiency was satisfactory.Secondly,It can help endoscopists to identify more early gastric cancer,especially for inexperienced endoscopists in primary hospitals.It can avoid missed diagnosis,shorten diagnosis time and improve diagnosis consistency.Moreover,it has good diagnostic efficacy for non-neoplastic lesions.We believed that the assisted diagnosis system for early gastric cancer will be applied broadly in the future.