| Background. Severity of illness prognostic models may be used in the intensive care unit (ICU) for benchmarking, quality improvement and external audit. It is essential that such models perform well. Within the past five years each of the major adult prognostic systems, namely, the Acute Physiology and Chronic Health Evaluation (APACHE), the Simplified Acute Physiology Score (SAPS), and the Mortality Probability Model (MPM) has been updated.1-4 There have not been comparisons between the new versions.;The presence of a "Do Not Resuscitate (DNR)" order is an independent predictor of mortality in ICU patients. Of the major ICU prognostic models only MPM0III includes DNR status as a predictor. The influence of DNR status on APACHE III and IV and SAPS 3 is unknown.;Hypothesis. We hypothesized that there would be differences in the performances of APACHE III, APACHE IV, SAPS 3 and MPM0 III. We further hypothesized that there would be differences in the performances of APACHE III, APACHE IV, and SAPS 3 when DNR status was included as a predictor variable.;Methods. A retrospective cohort study was performed. Demographic, physiologic and outcome data for 2596 patients admitted to one of three ICUs (medical, surgical, mixed) at our tertiary referral center in 2006 were collected. The presence or absence of a DNR order on ICU admission and at the end of the first ICU day was recorded. The performance of each of the four models, with and without inclusion of first day DNR status, was assessed using the area under the receiver operating characteristic curve (AUC) for discrimination and the Hosmer-Lemeshow statistic (HLS) for calibration. Discrimination was considered excellent, very good, good, moderate and poor with AUC values of 0.9-0.99, 0.8-0.89, 0.7-0.79, 0.6-0.69, <0.6, respectively. Calibration was considered poor with HLS p-value <0.05. Comparison of model performance was as described by Hanley.5 The Brier score was used to evaluate overall accuracy of the predictive models.;Results. Of the 2596 patients studied, 283 (10.9%) did not survive to hospital discharge. The areas under the receiver operating characteristic curves (AUCs) (with 95% confidence intervals, CI) of the models for prediction of hospital mortality were 0.868 (0.854-0.880), 0.861 (0.847-0.874), 0.801 (0.785-0.816), and 0.721 (0.704-0.738) for APACHE III, APACHE IV, SAPS 3 and MPM0III, respectively. The discriminatory performances of APACHE III and APACHE IV were similar (p = 0.621) and classed as "very good". SAPS 3 also had very good discrimination, but was inferior to APACHE (p<0.001). MPM0III had "good" discrimination and was inferior to the other models (p<0.001 for each comparison). The HLS for the models were 33.7, 31.0, 36.6, 21.8, for APACHE III, APACHE IV, SAPS 3 and MPM0III, respectively. Each of the HLS generated p values < 0.05. Brier scores for APACHE III, APACHE IV, SAPS 3 and MPM0III were 0.0771, 0.0749, 0.0890, and 0.0932, respectively.;DNR orders on ICU admission and at the end of the first ICU day were present in 211 (8.1%) and 252 (9.7%), respectively. Of the non-survivors, 19.4% had DNR orders on admission versus 6.7% of the survivors, p < 0.01. At the end of the first ICU day, 32.5% of non-survivors were DNR versus 6.9% of survivors, p < 0.01.When DNR status at end of the first ICU day was included in the models, the AUCs were 0.876 (0.855-0.897), 0.868 (0.846-0.891), and 0.816 (0.791-0.841), for APACHE III, APACHE IV, and SAPS 3, respectively. There were no significant differences between the discriminative ability of the models with and without DNR status (APACHE III p = 0.103, APACHE IV p = 0.145, SAPS 3 p = 0.072). The HLS for the models with DNR status were 29.3, 33.3, and 29.0 for APACHE III, APACHE IV and SAPS 3, respectively. Each of the HLS generated p values < 0.05. Brier scores for the models with DNR status included were 0.0697, 0.0700 and 0.0780 for APACHE III, APACHE IV and SAPS 3, respectively.;Conclusion. In our cohort of patients APACHE III and IV had similar discriminatory capability. Both discriminated better than SAPS 3, which was, in turn, better than MPM0III. The calibration of the models studied was poor. Neither the discrimination nor calibration of APACHE III, IV and SAPS 3 were significantly improved by the inclusion of resuscitation status at the end of the first ICU day in the prognostic models. The overall accuracy of the predictive models was better for models with more variables. |
