Effects Of Training On Raters Of NMET Writing Performance

Writing assessment requires subjective evaluations of writing quality by raters. Subjectivity in rating may inevitably introduce inter-and intra-rater inconsistency,thus it is difficult to attain high rating reliability. As a large-scale and high-stakes test, the results of the National Matriculation English Test (NMET) have affected so many high school graduates that it is important to ensure rater's fairness of its writing performance part. Some researchers abroad claimed that rater training is essential in achieving reliable ratings of writing performance. The effect of rater training in writing assessment has been studied abroad for many years, but there have been sketchy studies about the effect of NMET rater training process at home. The purpose of the study is to detect the effectiveness of NMET rater training process by exploring the differences of inter-and intra-rater consistency before and after training, and try to find some strategies to improve the training process.The thesis reviews the literature related to the present study. It gives an overview of the writing assessment and literature on rater training, rater training process and the viewpoints about the function of rater training. Many-facet Rasch measurement adopted in this study is well introduced and related researches are also presented.In this study, from hundreds of teachers,4 novice raters and 4 experienced raters are selected to rate the a sample of 20 compositions twice before and after rater training in the context of the administration of NMET in Shanxi province in 2009.A holistic scale for NMET writing assessment is used in the study, containing 5 levels of writing proficiency. FACETS,a multi-faceted Rasch analysis program is employed to analyze both pre-and post-training sessions of rating to explore the differences in rater severity, self-consistency among 8 raters and rater bias before and after training.Furthermore, a questionnaire is designed to examine the attitude of NMET raters towards rater training and ask for their suggestions on the improvement of training process.The findings of present study demonstrate that the training process surely affects the severity of individual raters.The spread of rater severities after rater training is to some extent reduced, and raters are more agreed with each other in their severity levels, but some differences among raters still exist. After the training process some individual raters become more self-consistent in their judgments, but there is little change in intra-rater consistency across the two sessions for the whole group.Experienced and inexperienced raters differ in their severity and intra-rater consistency before and after rater training. There is a little reduction of individual biases toward certain examinees after rater training. So the rater training process was not as effective as we have expected.The majority of raters are willing to accept rater training and believe that rater training before actual rating is very important. Almost all raters hold an optimistic view of rater training effect. Some raters put up their opinions and suggestions on the improvement of training process.Finally, based on the findings, implications and limitations of the present study are discussed and future research is suggested.