A Design And Validation Of Computerized Adaptive English Language Proficiency Test

With the fast development of the measurement theory and computer technology,the development and application of computer adaptive language test system(CALT)is the hot spot of language testing of both domestic and abroad research field.Comparing with traditional paper-and-pencil language test(PPLT)and ordinary computer-based language test(CBLT),CALT has many advantages,such as more user-friendly,more accurate,more flexible in ways of presenting test items,more convenient in ways of administrating test and scoring,and more efficient.The purpose of this study was to: 1)design a computerized adaptive language test(CALT)to assess grammar and vocabulary proficiency in English using mixed-format with dichotomous and polytomous item response theory(IRT)models,and 2)to investigate the validity of the CALT under the assessment use argument(AUA)framework.In the process of item bank construction,data of all English majors in China who took part in the TEM-4 from 1996 to 2008 were thoroughly analyzed with Bilog 2.0,AMOS 7.0 and SPSS 20.0 software.The responses were used for item calibration and differential item functioning(DIF)detection.Research methods include: 1)exploratory factor analysis(EFA)with SPSS 20.0,confirmatory factor analysis(CFA)with AMOS 7.0,to examine the uni-dimensionality assumption;2)examine the local dependent assumption with Bilog 2.0;3)use 2PLM to examine dichotomous items and use GRM and GPCM to examine polytomous items with Bilog 2.0;4)DIF test with Bilog 2.0 and SIBTEST.For the second research objective,there are three procedures: CALT design,simulation,processing and validation.In the design of CALT,arrange the sequence of cloze,grammar choice and vocabulary choice;use maximum information method(MI)to select items,and consider content balance and exposure control;use expected a posteriori(EAP)to estimate ability;use a combination of variable-length stopping rule and fixed-length stopping rule.In the simulation of CALT,the Firestar and R software are employed.In the processing and validation of CALT,analyses include: 1)T test of SPSS 20.0 and CFA of AMOS 7.0;2)SEM of AMOS 7.0 to examine the computer familiarity,TEM4 results and CALT results;3)analyses of models above with AMOS 7.0.This study has investigated the procedures used to develop a CALT designed to assess grammar and vocabulary proficiency in English with mixed-format,and examined the validity issues of the CALT within Bachman and Palmer's(2010)AUA framework.Major findings of the study are summarized as follows,in order of the three stages,namely,item pool construction,overall CALT design,and CALT validation.Theoretically,the present study,for the first time in the literature,fully investigates the construct validity of a CALT.Construct validity of CALTs has not been fully investigated before,possibly due to the fact that no consensus has been reached as to what the CALTs measure.Practically,the present study provides insight into the specific procedures that need to be followed in the development of CALTs and points out a few key issues,such as DIF detection,that were ignored in previous CALT development research.The trend of using computerized language tests in large-scale language assessment in the world,combined with the power of CALTs in discriminating test takers in an effective way,makes this a critical area of study from a practical standpoint.There are limitations in this study,which I hope could inspire other researchers in future study.For instance,the possibility of applying one of the multi-dimensional IRT models in item calibration could be further explored;more flexible tow-tier fullinformation item factor analysis model to calibrate the grammar and vocabulary sections simultaneously could be applied;future studies should also attempt to incorporate other variables as mediating factors of the influence of computer familiarity on test takers' performance in the CALT,etc.