| With the large-scale implementation of online rating for various English examination scoring,the organizations of English examinations at all levels are now moving towards standardization.Nowadays,the change of rating systems has ushered in more challenges,which have aroused increasing research attention at home and abroad.In the rating process,raters are central to the performance assessment,and raters’ perceptions,which can partly measure the psychological state and thinking logic of the raters when rating,will make essential contributions to studies related to rating quality.This thesis aims to provide a comprehensive picture of the factors that will affect raters’ rating quality from the raters’ perspectives,and propose practical improvements to corresponding factors in the current online rating scenario.To address the research objective,the present study puts forward three research questions:1)What are the factors that affect the quality of online rating from the perspective of raters?To what extent does each factor affect the rating process and quality? 2)How do raters with different demographic characteristics and professional backgrounds differ in perceptions of the factors affecting the online rating? 3)How do raters with different rating quality levels differ in perceptions of the factors affecting the online rating?In order to solve the research questions,this study adopts a mixed-method by using data collected from a comprehensive survey and four semi-structured interviews.The survey is conducted after a high-stakes English examination.Participants are 101 college teachers with teaching experiences in English subjects for at least one year.Data are collected through 43 questionnaire items under six original dimensions and the background information part of the questionnaire.Through exploratory factor analysis(EFA),the study determines several factors that may affect the rating quality.The semi-structured interview data can be a supplement to provide a deeper understanding of how these factors affect the raters’ thinking logic,rating strategies and rating process.When discussing the relationship between raters’ professional backgrounds and the factors,the t-test is used to identify the significance between the two variables.Meanwhile,the same method is also adopted to confirm the relationship between raters’ rating qualities and the factors mentioned above.This study analyzes the raw data through SPSS 23,all analyses lead to the results listed as follows:The present study identifies seven main factors that affect the rating quality from the raters’ perspectives: training,personal condition,environment and service,rating scale,the actual use of rating scale,sample paper,and supervision.The study then clarifies the professional backgrounds of the participants and concludes from t-test that people with different characteristics perceive variously towards certain factors: For experienced raters,training and sample papers are not that helpful and raters are less willing to follow the rating scale then inexperienced raters.It is worth noting that by summarizing the characteristics of the groups with different rating quality,the author finds raters with different rating quality levels show various perceptions to the above factors when rating.This paper makes contributions to enlarge the scope of relevant factors in online rating environment.Based on the findings concerning raters’ rating performance,this paper further deepens the scholars’ discussions of how factors affect the rating quality and then makes detailed supplements to theories related to raters’ thinking process.More importantly,the study provides constructive suggestions for the current English test rating system from the following four aspects: improvement of the trial evaluation mechanism,personal setting for raters,setting of mandatory breaks and suggestions of salary models,putting forward suggestions for rating quality improvement and online rating system management in a more practical way. |
PDF Full Text Request