Predictive performance of asthma quality of care measures---A comparison under skewed outcome and cost distributions

Objective. To evaluate potential measures of quality-of-care in asthma as predictors of subsequent emergency hospital care under skewed class and cost distributions.;Methods. The California Medicaid (Medi-Cal) claims data from January 2004 to December 2006 was analyzed. Patients with persistent asthma were identified using the Health Employers Data Information Set (HEDIS) criteria and were stratified into high (≥ 6 reliever medication claims) and low-reliever (< 6 reliever claims) cohorts. The dependent variable was the occurrence of an asthma-related hospitalization or emergency department visit. Asthma quality-of-care measures assessed include the HEDIS performance measure of any asthma controller fill, a controller-to-total asthma medication ratio ≥ 0.5, proportion days covered (PDC) ≥ 0.8, and the number of controller fills. Covariates included demographic variables, prior medication and healthcare service use, and comorbidities. The quality-of-care measures were evaluated cross-sectionally over a one year period as well as longitudinally using quarterly assessments. Models were estimated using logistic regression in the cross-sectional study, and generalized estimating equations (GEE) in the longitudinal study. The k-fold cross-validation procedure was used to assess the internal validity of the predictive models. The ROC Convex Hull method and decision curve analysis were used to account for unequal error costs and skewed outcome distributions.;Results. A PDC ≥ 0.8 and a controller to total asthma medication ratio ≥ 0.5 were associated with decreased risk of subsequent emergency hospital care in the cross-sectional and longitudinal studies. Any controller dispensing and number of controller fills were not consistently associated with decreased hospitalizations and may reflect underlying severity. An asthma controller/total medication ratio >=0.5 was the best predictor of asthma related emergency hospital care as it had the greatest AUC in both the cross-sectional and longitudinal studies; however, the predictive power was modest (<0.6 in all analyses). The predictive performance of the baseline covariates was significantly higher, with the AUC ranging between 0.7 and 0.8. No asthma performance measure improved the AUC when added to the baseline covariates. Application of the ROC convex hull method suggests that these models are better than the "do nothing" strategy only when the cost of false negatives exceeds the cost of false positives. Similarly, application of decision curve analysis yields the greatest net benefit at low probability thresholds, implying the greatest utility occurs when the cost of a false negative exceeds the cost of a false positive. The low incidence of emergency hospital in each quarter of the longitudinal study yields a minimum cost classifier with only marginally better performance than the strategy of "assume all are patients are negative".;Conclusion. Two potential quality-of-care measures, the asthma medication ratio and proportion days' covered, appear to be superior to the current HEDIS performance measure of any controller dispensing. However, prediction models that incorporate demographics and prior resource use data have significantly better ability to identify patients at high-risk for asthma-related emergency hospital care than the asthma quality-of-care measures alone. The low incidence of emergency hospital care suggests that identifying high-risk patients based on accuracy alone may result in suboptimal outcomes. The cost of false negatives must exceed the cost of false positives for these models to be useful.