Examining the experimental designs and statistical power of group randomized trials funded by the Institute of Education Sciences

In federally sponsored education research, randomized trials, particularly those that randomize entire classrooms or schools, have been deemed as the most effective method for establishing strong evidence of the effectiveness of an intervention. Consequently, there has been a dramatic increase in the number of federally funded group randomized trials. However, the occurrence of a group randomized trial does not guarantee high-quality evidence. Low statistical power in group randomized trials may prevent studies from yielding conclusive evidence of the effectiveness of a program. In this study, I examine the experimental designs and power analyses for the group randomized trials funded by the National Center for Education Research (NCER) and the National Center for Education Evaluation and Regional Assistance (NCEE), two branches of the Institute of Education Sciences (IES). First, I categorize the studies by experimental design. Second, I calculate the minimum detectable effect size (MDES) for each study under plausible assumptions about intra-class correlations (ICC's), covariate-outcome correlations, and explanatory effects of blocking. Finally, I compare the MDES I calculate, denoted the recomputed MDES, to the MDES stated in the proposal.;The blocked designs, that is the three-level multi-site cluster randomized trial and the four-level multi-site cluster randomized trial, constitute the most common experimental designs. The MDES for the NCER studies ranges between 0.20 and 1.0. However, the less precise NCER studies, those with an MDES ranging from 0.40 to 1.0, tended to be funded prior to 2005. The NCER studies funded in 2005 and 2006 were more precise, generally with an MDES ranging from 0.18 to 0.40. This is a positive finding and shows significant growth and improvement in the precision of NCER studies over time. The NCEE studies also tended to be more precise, with all of the MDES's falling below 0.40. The comparison between the stated MDES and the recomputed MDES for the complete sample reveals that in 56% of the studies, the two MDES's are in the same range. However, a subgroup analysis shows that there is greater agreement between the recomputed and stated MDES's for the NCEE studies than the NCER studies.