A Study Of L2 Writing Rater Cognition From The Rating Experience And The Syntactic Complexity Perspective

Validity is the most fundamental consideration in language assessment concerning the score interpretation.However,the rating process is a complex cognitive process which is influenced by both internal factors of the raters and various external factors,and thereby affecting the test validity.This study aims to investigate the impact of rating experience and syntactic complexity on raters cognition and further on the rating quality.To achieve this goal,in the context of the TEM-4 integrated-writing task,this study explores the rating behavior and decision-making process of five novice raters and five experienced raters in scoring two texts with different syntactic complexity.All raters have gone through a training session and are required to make a judgement from summary,comment,organization and language use.By means of eye-tracking methodology,this study probed into raters’ scoring foci and text-rubric-scoring table interaction and their decision-making process in depth,combined with stimulated recall interview and semi-structured interview data.The main research findings indicated that there was a relationship among syntactic complexity,rating experience and rater cognition.In terms of syntactic complexity,raters would spend longer time and visit more on comment and organization in rating text with high syntactic complexity,fixate less on summary,comment and mechanics,and skip more in general.As for scoring rubric and table,raters tended to pay less attention to them in text with high syntactic complexity.In addition,the phenomenon of a similar number of fixation transitions among the three parts appeared.From the perspective of rating experience,the total rating time,fixation duration and visit count of novice raters were all higher than that of experienced raters in five categories.They also skipped less.Furthermore,novice raters focused and visited more on the scoring table and rubric and kept larger fixation transitions.However,experienced raters seldom referred to the rubric.Finally,there was no interaction effect between syntactic complexity and rating experience on rating behavior.As for the decision-making process,the results reflected syntactic complexity exerted no influence.In terms of rating experience,both cognitive and metacognitive process of two groups were revealed,while a more complex decision-making process was revealed among experienced raters.Furthermore,it can be summarized that there was no correlation between eye movement metrics of summary,comment and organization with corresponding rating discrepancy except for language use.Taken together,this study employed eye-tracking method in rater research and investigated rater cognition in depth for the purpose of providing a better interpretation of final scores and cognitive evidence for integrated-writing task validity.