A Comparative Study Of Teachers-as-Raters And External Raters In EFL Writing Assessment: A Case Study

In writing tests in the EFL classroom, the students’ performance is typicallyrated by their own teacher, who has built a rapport with the students after teachingthem for a period of time. In this context, it is inevitable that the teacher’s ratingbehavior will be influenced by classroom-related background factors, such as theteacher’s expectations, the syllabus, and teaching experience. It is thus interesting tocompare the teacher’s behavior in rating the writings of his own class, when he hasbuilt a rapport with his students, and his behavior in rating the writings of anotherclass, with whom he has no rapport. The potential effect of the teacher-studentrapport on the teacher’s rating behavior will provide further insight into how raterbackground brings about rater variability, which will then be interpreted from theangles of both summative and formative assessment.To understand the potential effect of the teacher-student rapport on the teacher’srating behavior, an exploratory study was conducted in the context of a weekly testof English in two senior classes (N=60) in a senior high school in China. Whilerating the writing samples of their own class as usual, the EFL teachers of the twoclasses were asked to rate the writing samples of the other class as well. Theteachers also commented on the writing samples in the rating process, and wereinterviewed after the rating about their knowledge and expectation of their students,their instructional emphases，as well as their beliefs about writing construct.The mean ratings given by the two raters to the two classes were comparedthrough Mixed ANOVA. The results indicated a significant main effect of raters, butnot of classes. No significant interaction between raters and classes was found. Bothraters gave higher scores to their own class than to the other class. The raters’comments were coded by two independent coders into four major categories: language, content, organization, and mechanism. The chi-square tests ofindependence of the frequency distribution of these comments indicated a significantthree-way association between classes, teachers, and categories. One teacherconsistently emphasized organization while the other played down the importance oforganization. Subsequent interviews showed that when the students’ identity wasknown, both teachers tended to give higher scores to their own students in order to“encourage” them. Furthermore, the emphases on different categories in the ratingprocess reflected the general beliefs of the teachers about the writing construct andin their instructional practices. In summary, teacher-class rapport was found to havean effect on both the process and product of essay rating in the EFL classroom.The qualitative analyses were then discussed in the argument-based frameworkof validity, which revealed some major concepts concerning the teachers’considerations of the four validity categories: construct validity (CV),relevance/utility (RU), value implication (VI), and social consequences (SC). Interms of summative assessment, the differences attributable to teacher-class rapportcould be interpreted as construct-irrelevant variance. However, from the perspectiveof formative assessment, the teachers’ rating practices are justified as part of theirteaching practices，aiming at improving the learning context and providing theeffective feedback.