Building evidence for the evaluation of English learners' writing scores

Performance-based assessment (PBA) has become the primary means for assessing L2 learners' writing abilities. Despite its popularity in L2 writing, validation is a contentious issue in PBA, as scoring rubrics are rarely entered as evidence for the appropriateness of decisions made from test scores (Cumming et al., 2004; Leung & Lewkowicz, 2006). In an argument-based approach to validity, the investigation of scoring rubrics is an important aspect of a major inference, evaluation. Evidence that supports the evaluation inference is crucial in the validation process, since decisions will likely be less valid if scoring rubrics are not adequately constructed and appropriately used (Kane, 2006). To provide evidence for the evaluation inference, five assumptions should be supported: (a) rubrics capture relevant aspects of performance at different score levels; (b) attempts were made to standardize scoring procedures; (c) raters know how to implement rubrics; (d) experienced and novice raters score students' writing similarly; and (e) rubrics function appropriately with their existing scales.;This research study was conducted in 2009--2010 whose purpose was two-fold: (1) to investigate the quality of scoring rubrics used to assess L2 students' writing ability at four intensive English programs (IEPs) and (2) to determine the value of a proposed framework for examining the evaluation inference. The study incorporated a multiple case-study methodology, whereby quantitative and qualitative evidence was collected. The results indicated that rubrics captured relevant aspects of writing performance and that the rating scales used in the rubrics functioned appropriately to distinguish different levels of performance, despite the finding that teacher-raters could have benefitted from additional scorer training. Also, the results of the investigation of the evaluation inference demonstrated that IEP administrators perceived the framework as being coherent and useful, but that it could have been more adequate and implementable.;The implications support the view that (a) scoring rubrics should be easier to implement for IEP teachers and students, (b) mixed-methods research is a complement to quantitative and qualitative research alone, and (c) the evaluation inference framework serves as a model for justifying the decisions that are made based on writing scores.