Assessing the consequences of response probability choice on student performance across subpopulations

The purpose of this study was to examine the effect that four different response probability criteria (RP50, RP67, RP75, and RP80) have on achievement level categorization for an archival data set from the Kansas Assessment Program.;Data from four grade/subject situations (4th grade mathematics, 5th grade reading, 7th grade mathematics, and 8th grade reading) were used to increase the generalizability across grades and content areas. Cut scores for each response probability were established using a Rasch model to transform percent cut points onto a theta scale and back after cut point calculation. These values were transformed back to the raw score scale so that each examinee's score could be categorized into an achievement level based on the response probability criterion.;Multinomial logistic regression analyses were conducted for the overall student populations in each grade and the populations disaggregated by two demographic variables, race/ethnicity status and socio-economic status. The results showed that the achievement gap between White students and Black students decreased if RP67 was used instead of RP50 in the two mathematics data sets. This was not the case for the achievement gap between Hispanic students and White students, nor was it the case in the reading data sets. Across all four data sets the achievement gap between those students receiving lunch services and those not receiving lunch services was less when RP67 was used.