A Study On Rater Bias In Rating The NMET Writing Performance

This paper aims to investigate the severity or leniency of raters when they are rating NMET writing papers and the consistency among and within raters by adopting the Multi-Facet Rasch analysis. Moreover, it also aims to detect rater bias toward candidates, traits (content, grammar, vocabulary, skill and structure) and score sections on NMET rating scale through bias analysis, in other words, rater vs candidate, rater vs trait and rater vs scale bias analysis.The author randomly chose a sample of 10 writing compositions from the archives of freshmen in one university, Shanxi province. Then, each rater was asked to rate each paper twice, first in terms of NMET rating scale, and then in terms of the five-trait scale. We have the following main findings from our data analysis:There are significant differences in the levels of severity of raters. Among the 10 raters, Rater 9 was the most severe one and Rater 10 was the most lenient one with the NMET rating scale; while in terms of the five-trait scale, Rater 1 was the most severe one and Rater 10 was the lenient one.Not all of the raters could keep self-consistent when they were rating and raters behaved inconsistently in their rating on different traits.Raters tended to be more biased toward candidates with relative high writing ability.Different raters exhibited different bias patterns toward different traits.Raters exhibited bias patterns toward score sections on NMET rating scale.The results of this study could help to improve the reliability of raters in their rating and provide feedback for raters' selection and training, and according to rater vs scale bias analysis, it can help modify and improve rating criteria. At the same time, the present study demonstrated that the Multi-Facet Rasch Model is a powerful tool in monitoring the rating quality and in writing assessment.