Development And Validation Of Machine Learning Prediction Models For Postoperative In-hospital Mortality And ICU Admission

More than 300 million surgeries are performed worldwide each year.With the improvement of surgical techniques and perioperative management,perioperative mortality has gradually declined in recent years,but the current global perioperative mortality rate is still between 1-4%.Differences in medical procedures may lead to different outcomes for high-risk surgical patients,in which ICU admission is an important link.More accurate prediction of patients’ risk of adverse outcomes after surgery could assist anesthesiologists and surgeons in making clinical decisions.Machine learning algorithms have shown superior performance over traditional methods in a variety of scenarios,including the perioperative field.We will use local data sets established through feature engineering to predict postoperative in-hospital death and ICU admission using machine learning algorithms based on preoperative,intraoperative,and postoperative information at different time points.Methods:This study is a real world data mining study,mainly relying on the database of surgical anesthesia system and electronic information management system of West China Hospital.In this study,the ids of all patients with surgical anesthesia records in the surgical anesthesia system of West China Hospital from the test run time of the anesthesia system to November 2020 were screened and returned to the hospital information management system for query with the index of patient IDS.All inpatients undergoing surgical and interventional surgery requiring anesthesiologists to perform anesthesia or supervision from the test run of the anesthesia system in West China Hospital of Sichuan University to November 2020 were retrospectively collected.With the patient ID as the main key,the perioperative information of patients stored in various systems,such as demographic baseline,complications,laboratory examination and surgical anesthesia information,was reassembled into a complete data set for analysis by Map Reduce method.The primary outcome was in-hospital death after surgery.The secondary outcome of this study was postoperative ICU admission.After identifying the structure and content of the database and clearly extracting the data type,the data set is structured and preprocessed by feature engineering.A total of 289 input features were determined for the preoperative model,including patient baseline characteristics,Healthcare Cost and Utilization Project(HCUP)surgical type,anesthesia mode,complications,preoperative blood test results,imaging reports,surgeons and anesthesiologists.A total of 516 input features were identified for the perioperative model,including preoperative,intraoperative and early postoperative features.Mutual Information Quotient(MIQ)is as the degree of importance of features.Feature selection was carried out through a two-step method.The MIQ of input features was arranged from large to small,and the backward iteration was carried out in 50 dimensions until all features were included,and the subset with the best performance was selected as the final feature set.Four machine learning algorithms are used in this study: Support Vector Machine(SVM),Random forest(RF),Adaptive Enhancement algorithm(Ada Boost),Artificial Neural Network(ANN,Artificial Neural Network).In this study,these four algorithms were also used to build Ensemble models using Platt Scaling.The hyperparameters of each algorithm are determined by empirical method and repeated test method.The evaluation indicators of this study included sensitivity,specificity,accuracy,receiver operating curve(ROC),Receiver Operator curve,F1 score,positive likelihood ratio,negative likelihood ratio and positive predictive value Predictive value,Negative Predictive value,Diagnostic odds Ratio,and Brier Score.In this study,stratified Shuffle K-fold cross validation will be used to verify the model performance.All samples will be randomly divided into 10 sets,5 for training sets,2 for validation sets and 3 for test sets.Validation set data is used to adjust hyperparameters,set classification threshold and probability calibration,training set data is used to train the model,and test set data is used to evaluate the model,so as to reduce the risk of overfitting.In the establishment of a simple traditional statistical model,initial screening of model variables was carried out through MCID(minimal clinical important difference,minimum clinical significance change value).Subsequently,dimensionality of model variables can be reduced layer by layer through LASSO(Least Absolute and Selection Operator),collinearity test between variables,monotonicity test between variables and outcomes,and Step AIC.The final variables were modeled by Logistic regression.This study will be divided into groups according to their outcomes.χ2 test or Fisher’s exact test was used to compare categorical variables,and T test or MannWhitney U test was used to compare continuous variables.The standard mean difference(SMD)is used to measure the difference between groups.P<0.05 was considered statistically significant.Results:A total of 165801 patients were enrolled in this study,of which 1741(1.05%)died in hospital,including 758(5‰)died after elective surgery and 983(8.8%)died after emergency surgery.A total of 15,060 patients were admitted to ICU after surgery,including 12,091 patients admitted to ICU after elective surgery,with a rate of 7.8%,and 2969 patients admitted to ICU after emergency surgery,with a rate of 26.6%.The prevalence of postoperative complications,such as anemia,hypertension,diabetes,chronic obstructive pulmonary disease,liver disease,and chronic heart failure,was higher in patients who died after surgery and those who were admitted to ICU after surgery than in normal discharged patients and those who were not admitted to ICU.The differences of hemoglobin,white blood cell count,neutrophil percentage,lymphocyte percentage,albumin,glucose,creatinine,serum calcium and other biomarkers in preoperative blood tests between patients who died after surgery and those who were discharged from hospital normally are of obvious clinical significance.The difference in preoperative test results between patients admitted to ICU and those not admitted to ICU was similar to that between patients who died and those who were discharged normally,but the difference was smaller.Compared with normal discharged patients,the time weighted average of intraoperative center rate and pulse rate was significantly larger in the patients who died after surgery,and the time weighted average of intraoperative Peak value was relatively higher.The proportion of intraoperative blood products was significantly higher,and the amount of intraoperative in and out was larger,among which the difference of colloidal fluid was more obvious than that of crystal fluid.Intraoperative blood gas analysis showed that compared with normal discharged patients,dead patients had lower hemoglobin minimum value,higher lactic acid accumulation,and were more prone to acidosis.Among the machine learning algorithms for predicting postoperative hospital death and postoperative ICU admission based on preoperative features,the OVERALL AUROC and AUPRC of SVM,Ada Boost and RF algorithms increased with the increase of the included features.When all features were included,the AUROC and AUPRC of the three machine learning algorithms were the largest.The AUROC of SVM,Ada Boost,RF and ANN models were 0.9465,0.9363,0.9453 and 0.9305,and the AUPRC were 0.4454,0.3989,0.5286 and 0.4844,respectively.The corrected Brier Score was 0.0085,0.0082,0.0068 and 0.0073,respectively.The AUROC,AUPRC and Brier score for predicting postoperative death were 0.9516,0.5371 and 0.0075,respectively.The AUROC of SVM,Ada Boost,RF and ANN prediction models for postoperative ICU admission were 0.9702,0.9790,0.9826,0.9720,and AUPRC were0.8742,0.7984,0.9096,0.8894.Brier score after adjustment was 0.0283,0.0334,0.0193,0.0213.Ensemble model predicted that the AUROC,AUPRC and Brier score of postoperative ICU admission were 0.9830,0.9124 and 0.0222 respectively.In the prediction model of postoperative mortality risk based on perioperative features,when the top 500 features were included,Ada Boost had the highest AUROC(0.9508),AUPRC(0.4476)and Brier score(0.0083)after correction.When the top 500 features were included,RF had the largest AUPRC of 0.5921,AUROC of 0.9595,and Brier score of 0.0066 after correction.When SVM included the first 500 features,AUROC was the largest,AUROC was 0.9545,AUPRC was 0.4711,and Brier score after correction was 0.0084.When the top 500 features were included,ANN’s AUPRC,AUROC and Brier scores were 0.4979,0.9377 and 0.0071,respectively.The AUROC,AUPRC and Brier scores of Ensemble model were 0.9641,0.5891 and0.0068 respectively.A total of 13 variables were included in the simple prediction model of postoperative death,with AUROC of 0.9412,AUPRC of 0.3429 and Brier Score of 0.0082.A total of 14 variables were included in the simple prediction model for postoperative ICU admission,with AUROC of 0.9351,AUPRC of 0.6382,and Brier score of 0.0479.Conclusion:This study,based on a large sample of real-world data from a single center at West China Hospital of Sichuan University,confirmed that postoperative death is still not an uncommon serious postoperative adverse event,and that postoperative mortality and postoperative ICU admission rates at the center were similar to those reported in similar studies.There were significant differences in the distribution of a number of indicators between patients who died after surgery and those who were discharged from hospital normally.In this study,289 preoperative characteristics were collected,and four machine learning algorithms including SVM,Ada Boost,RF and ANN were used to successfully establish the prediction model of postoperative death and postoperative ICU admission based on preoperative indicators.The verification results showed that: SVM had the best distinction in predicting postoperative death and RF had the best distinction in predicting postoperative ICU admission.Although the mathematical basis of the predicted scores is different among the models,the scores are well correlated.The Ensemble model,which was built by combining multiple machine learning models based on different mathematical foundations with Platt Scaling,achieved higher prediction discrimination than a single machine learning algorithm.After the inclusion of new information,the predictive performance of the new model was improved compared to the death model based on preoperative information only.This research used local data sets from West China hospital through the combination of traditional statistical methods and clinical knowledge,cautiously discriminating the relationship between variables and outcomes,selecting a group of predictive variables,and use these variables to establish the prediction model,making the nomogram of the model and web tools,The simple statistical model finally established has good performance in predicting postoperative death and postoperative ICU admission.