Probing Into The Relationship Between Raters And Rating Scale In The NMET Essay Rating

The assessment of EFL writing has long attracted the attention of those researchers in language testing field and teachers who work on the instruction of EFL writing, because it needs raters to make subjective decisions based on a rating scale, which brings about some great controversial problems to language testing field.The rating process of English essay rating involves rater and rating scale. In the language testing field at home, there remains much scope in the research into rating process, and the researches about the relationship between rater and rating scale in the writing assessment are rarely done, especially the systematic empirical investigation of the relationship in NMET essay rating.The study in this thesis aims to investigate the relationship between raters and rating scale in NMET essay rating empirically, and explores the importance of rater and rating scale in NMET essay rating. Specifically, the present study is to investigate how raters interpret and use rating scale, how different rating scales influence rating behaviors and the rating outcome through comparison of the effects of holistic rating scale and analytic rating scale. Finally the necessity and importance of rater training are demonstrated and other ways to improve the rating reliability are found out.The present study analyzes the relationship between rater and rating scale in NMET essay rating qualitatively and quantitatively. In the early period of NMET rating session in the summer of 2009, a questionnaire including 8 questions was used to investigate the NMET raters'opinions about the NMET essay rating. After collecting all the questionnaire paper, the author analyzed and summarized the responses from raters. According to the outcome of Question 5 in the questionnaire and the holistic rating scale used in NMET essay rating, the author designed an analytic rating scale for future use. From the rater responses of questionnaire,8 raters were selected randomly to rate the 10 writing samples holistically and analytically. According to the outcome from Rasch analysis, the author compared the raters'decisions based on two different kinds of rating scales.The results from the study show that there is difference in raters'interpretation and use of rating scale, which may lead to the differences in rating outcome. The results also indicate that, in the NMET essay rating, most raters award scores by virtue of the rating scale, different rating scales may lead to different rating behaviors of raters, and finally the two factors may influence the outcome of rating, and even with the same rating scale, raters may differ in the interpretation and use of rating scale, too. It is demonstrated that the holistic rating scale is more appropriate for the NMET essay rating because of its higher reliability. As regards the inconsistency of raters, rater training and better specification of the rating scale are suggested in this thesis.