Establishing Comparability Between Collaborative And Self-paced Rater Training Methods

Rater training is integral to the scoring of language performance assessments.While research has generally shown that training is conducive to reducing rater variability and thus ensures the quality of test scores,there has been a paucity of research into the relative efficacy of different training methods,especially when new raters are involved.The study adopted a measurement perspective and aimed to systematically examine and compare the effects of collaborative and self-paced training methods on the scoring performance of newly-recruited raters in the context of the ETIC test.The study simulated authentic rater training sessions in which twelve university English teachers who had no prior experience with the ETIC test scoring participated as new raters.The raters were assigned to either collaborative or self-paced training group,both of which were convened by the same trainer.China Language Assessment provided the training manual which contained a prompt of the ETIC Intermediate Writing Task 3,the analytic rating scale,and thirty scripts.Both training methods involved the same training materials and a series of activities including familiarization,practice,comparison,discussion,and feedback.The only difference was that the raters in the collaborative group studied and discussed together under the instruction of the trainer while the raters undergoing self-paced training studied independently and discussed their progress only with the trainer.The raters then proceeded to score an additional package of eighty scripts which had been pre-scored by a panel of expert raters.The resultant rating data were used to create another set of accuracy rating data by comparing with expert ratings using a dichotomous scoring method.The two sets of data were analyzed separately with the Facets program under two different many-facet Rasch models for rater invariance and rater accuracy.Results of rater calibrations,model-data fit,and interactions showed that all individual raters met the requirements for operational scoring,and that differences in scoring performance between the two groups were not significant.It was therefore concluded that both training methods were equivalently effective on the scoring performance of the twelve new raters.The results highlighted the comparable effectiveness of self-paced and collaborative training methods on rater performance,with special reference to new raters.The research findings carry implications for the development of rater training programs and the validity argument of the ETIC test.Future research can compare the effects of the two training methods on rater cognition and behavior from the cognitive perspective in order to complement the findings of this study.