A Comparative Study Of Analytic And Holistic Approaches For The Computerized Oral Test Of MET

In the field of language testing, a substantial body of work has been produced dealing with the debate between holistic and analytic rating rubrics for speaking tests. Despite this, the effects of these two different scales on outcomes are still not well understood, especially in the case of speaking assessment. For that reason, the current study attempts to explore empirically whether the application of these alternative scales may cause any difference in the scores allocated by the same raters and the way the raters rate the spoken texts. The context is the Computerized Oral Test of Guangdong Matriculation English Test (MET), which is used by institutions of higher education in China to assist in matriculating English majors. Such a test, as its name suggests, is administered via computer technology. Those who take part in this test are supposed to sit in a language lab equipped with computers. Altogether ten instructors from four universities in Guangzhou took part in the study, providing results of two questionnaires as well as 600 scores for 30 speech samples produced by those who had participated in the COT of Guangdong MET in 2007.Data analysis was approached in this study from two complementary directions. Qualitative study was conducted to analyze the results of two questionnaires, which were designed to elicit the raters' attitudes and opinions towards holistic approach, analytic approach as well as the corresponding training sessions. Statistical analysis and FACETS analysis, which provides estimates of rater severity on an equal-interval scale as well as fit statistics, were drawn on to examine reliability, rater severity and rater-ratee interactions in relation to the two rating settings. The qualitative and quantitative analyses resulted in the following findings: First, both types of rating scales were reliable and could produce reliable results. Second, with regard to rater severity, the differences in severity estimates were more pronounced in the analytic ratings although both holistic approach and analytic approach provided consistency and significant differences in severity among the raters. Third, there existed no significant rater-candidate interactions in the holistic scorings. However, seven raters were found to score certain candidates more leniently or harshly using the analytic rating scale. What's more, the patterns of rater-candidate interaction were not very clear, with each rater having his/her unique bias pattern.Based on the data collected and preceding findings, it might be concluded that holistic approach is more appropriate in such a high-stakes assessment program if detailed information of candidates' speaking ability is not required.