A FACETS Analysis Of Rater Bias In Measuring Chinese Students' English Writing

This article describes a study conducted to explore differences in mean scores the test takers received in two rating sessions, rater severity and rater-candidate interactions among four raters, by comparing two different types of scoring methods: holistic scoring and analytic scoring. The participants in this study consisted of 4 raters and 37 university students. The statistical analyses show that both types of rating scales are reliable and can produce reliable results, though there are some differences in the mean scores across the two rating sessions. Since the reliability coefficients of the two kinds of ratings are close, we are not sure whether the analytic rating scale is significantly more reliable than the holistic one without statistical analysis.FACETS analysis, which provides estimates of rater severity on a linear scale as well as fit statistics, on the other hand, demonstrates that for the four raters in the holistic scorings, rater severity is clustered around the mean. In addition, there was no significant rater-candidate interaction. In the analytic ratings, however, the differences in severity estimates are more pronounced, with significant differences found among the raters;and some raters scored certain candidates more leniently or harshly. Nonetheless, the patterns of rater-candidate interaction are not very clear, with each rater having his/her unique bias pattern. The conclusion of this study is that holistic scoring is more appropriate even in high-stakes assessment program, if detailed information of candidates' writing ability is not required.