A Study Of TEM4-oral Validity: A Test-taking Process Approach

Since the implementation of Test for English Majors-Band Four Oral Test (Hereafter TEM4-Oral) in 1999, it has become a crucial criterion to evaluate the spoken English proficiency of English majors. Now over 30,000 students sit for this nationwide test every year. The test scores are used not only by teachers and schools to evaluate the candidates'English speaking proficiency and diagnose problems in English learning, but also by the society to judge the candidates'ability in future jobs. To ensure the appropriateness of inferences and actions based on test scores, the validity study has become important and necessary. Researchers have drawn attention to the validation of TEM4-Oral, and collected empirical evidence based on statistical analyses of test content and rating processes. Yet validation is an ongoing process, which involves collecting different sources of empirical evidence. There are five sources of validation evidence: the content relevance; the relationships between responses to different test items; statistical analyses of the internal structure of the test; relationships between the test and measures of other construct; the washback of the test. Among these five sources of evidence, the relationships between responses to different test items include evidence based both on the raters'assessment process and on the test-takers'test-taking process. Some researchers had collected validation evidence from raters'assessment process. However, up to now, there is still a lack of validation evidence based on test-taking processes.This study aimed at validating TEM4-Oral of 2010 from a test-taking process perspective. Twenty sophomores of English majors from a university in Suzhou participated in the study. The instruments of stimulated recall and interview were used to elicit the participants'test-taking processes. The competences that were reported to be used in their test-taking processes were then compared with the competences that the test is intended to measure. The qualitative data analysis yielded two major findings: (1) all the competences that the test is intended to measure were used by the test-takers in their performance, which proved a full coverage of the construct; (2) some construct-irrelevant components were also reported by the test-takers, which threatened the construct validity. Finally, theoretical, practical and methodological implications of the study were discussed.