The effectiveness of test-takers' participation in development of an innovative Web-based speaking test for international teaching assistants at American colleges

The purpose of this study was to investigate test-takers' behaviors and contributions by empowering the test-taker developers in language test development process. The test development team consisting of teachers and test-takers proposed a democratic language assessment and a validation model for the development. The testtakers participated in developing a web-based speaking test for international teaching assistants at American colleges and they articulated and reflected their needs into the process interacting with external and internal stakeholders. The audited specification-driven testing framework was adopted in the ADDIE model for the development. Validity arguments were gathered and justified from the beginning to the end of the test development. The effectiveness of test-takers' participation in the test development was investigated (a) the needs analysis on the SPEAK and its reflection in the test, (b) the test takers' roles in their interactions and power relations with regard to their contributions to task and validity arguments constructions, and (c) their contributions in validation studies with the data collected from a field test.; The results of content analysis of the SPEAK showed the mismatch between the purpose and its use in a local context and the findings from the needs analysis were aligned with the team's decisions on test design considerations reflected into the development. The results of the group dynamics and audit trail supported positive contributions of the test-takers to the test development by revealing their roles, interactions, and power relations. The results of the validation studies with the data from test scores and survey and interviews also supported the effectiveness of the test-takers' participation in the test development by indicating their equal contributions to task construction, significant rating difference of one rater and rating discrepancy on the tasks in certain situations, no misfit across rater and task, and the need of more rater training. It could be concluded that the test-takers' participation in language test development was supportive results of validating test development process as well as the test product. Through the process, authenticity of topics, content, and other test conditions were achieved in their local academic contexts.