Evaluation Of Detection Performance Of Artificial Intelligence-aided Diagnostic Systems For Pulmonary Nodules

PurposeTo investigate the effective dose(E)and convolution kernel’s effects on the detection of pulmonary nodules in different artificial intelligence-aided diagnostic systems(AIADSs).And to evaluate the performance of four systems in identifying and measuring four types of pulmonary nodules.MethodsSimulated nodules of various types(small ground-glass nodule[SGGN],small solitary nodule [SSN],GGN,and SN)in the Lungman phantom were CT scanned at different levels of E(3-5,1-3,0.5-1,and< 0.5 m Sv)and all the images were reconstructed with different convolution kernels(B30f,B60 f,and B80f).The number of nodules and corresponding volumes in different images were automatically detected by four AIADSs(A,B,C,and D).Sensitivity,false positives(FPs),false negatives(FNs),relative volume error(RVE),and missed detection rate(MDR)for different types of nodules of each AIADS under different E and convolution kernel conditions.And the performances of systems in identifying four types of nodules were also evaluated.ResultsSystem B had the highest median sensitivity(100%).The median FPs of systems B(1)and D(1)was lower than A(11.5)and C(5).System D had the smallest RVE(13.12%).When the E was <0.5 m Sv,system D’s sensitivity decreased,while the FPs and FNs of systems A and B increased significantly(P < 0.05).When the kernel was changed from B80 f to B30 f,the FPs of system A decreased(33.57%),while that of system C increased(58.84%),and the RVE of systems A,B,and C increased(P < 0.05).The RVE of system D was the lowest for all nodules,except for SGGNs.System B had the smallest RVE for SGGNs.In the Bland-Altman test,only systems B and D passed the consistency test(P = 0.40).In addition,the MDR of system B for SSNs,GGNs,and SNs was 0.00% and for SGGNs was 4.17%.The system A’s MDR were the highest with 71.30%,25.93%,and 47.22% for SGGNs,SSNs,and GGNs,respectively.Receiver operating characteristic curve analysis indicated that system D had the best performance in recognizing SSNs and GGNs,with areas under the curve(AUC)of 0.91 and 0.68.System B had the best performance for SGGNs(AUC = 0.91).ConclusionSystems B and D have high detection efficiency under normal or low dose conditions and show better stability.However,the detection efficiency of systems A and C would be affected by the E or convolution kernel,but the E would not affect the volume measurement of four systems.Among the four types of nodules,SGGN is the most difficult to identify;System D is the most accurate in measuring volume,and system B is the most accurate in identifying nodules.However,the accuracy and precision of AIADS still need to be further improved.