Student achievement in a publicly funded voucher program: A comparison of three statistical techniques for analyzing and interpreting data

The current study examined data collected on a longitudinal cohort of students as part of the Cleveland Scholarship and Tutoring Program (CSTP) from the beginning of first grade through the end of fifth grade. Three models (repeated measures analysis of covariance [RM-ANCOVA], ordinary least squares [OLS], and hierarchical linear modeling [HLM]) were compared to address the research question: Do the models differ in the inferences that can be made regarding the impact of using a voucher to attend private school on reading and mathematics achievement growth? Models included measures of achievement, voucher group, minority status, and teacher experience. Two OLS and HLM models were analyzed: one model included students with complete data across all variables (OLS-1 and HLM-1) and the second model included students with some missing data (OLS-2 and HLM-2). Effects for voucher group on mathematics achievement were found for each model. Non-voucher students were found to have higher fifth grade achievement than voucher students (OLS-1, OLS-2, and HLM-1) and had higher yearly growth in achievement than voucher students (HLM-1). Voucher students outperformed non-voucher students in the fall of first grade, but these differences disappeared by the end of first grade, and non-voucher students had higher mathematics achievement than voucher students each year thereafter (RM-ANCOVA). Minority status (RM-ANCOVA, HLM), teacher experience (both OLS models), and first grade achievement (both OLS models) also were significant predictors of mathematics achievement. The results for reading were characterized by a lack of effects, especially for the voucher group. Effects of using a voucher were found only in the OLS-1 model, with non-voucher students having higher fifth grade achievement than voucher students. Minority status and first grade reading achievement also were significant predictors of reading achievement. Overall, the models were relatively consistent regarding the impact of using a voucher. HLM offered advantages over the other models for analyzing the impact of a voucher on student achievement. However, there was some evidence to suggest that, at least for mathematics, OLS could be used despite the data being multilevel in nature.