Rater variability and reliability of constructed-response questions in high-stakes tests of English language arts and mathematics: Implications for educational policy

This study used generalizability (G)-theory as a theoretical framework to study the effects of multiple facets on the variability and reliability of New York State Grades Four through Six constructed-response scores for ELA and Mathematics. The impact of the study could be both timely and significant given the current political climate surrounding accountability in New York State. Specifically, the study examined the impact of scoring method (holistic and analytic) used, gender, teaching experience, assessment rating experience, and certification of the rater on the score variability and reliability of the constructed-response tasks.