The Analysis And Comparison Of Item Selection Strategies In Computerized Adaptive Testing

Examinations are very good ways of evaluating teachers’teaching quality andstudents’knowledge skills. At present, the traditional paper-and-pencil test form isused mostly in homeland tests. And the paper-and-pencil tests use the same items forall examinee, so it does not provide items with the corresponding difficulty level tothe examinees according to their real ability, besides items too difficult or too easycould lead to meaningless test results, so paper-and-pencil tests can’t measure the realability of the examinees precisely. However computerized adaptive testing(computerized adaptive testing, CAT for short) controls testing items of everyexaminee by the adaptive item selection strategies, which means different examineecan have different items of different levels of difficulty according to his ability, henceCAT can measure examinee’s real ability level at higher accuracy. Now CAT is usedwidely in foreign countries, such as the GRE, TOEFL, the GMAT and so on.Compared with the traditional paper-and-pencil test, CAT has new features:Firstly, CAT has adaptive item selection procedure; secondly, CAT does not limit totime and geography. As the central part of CAT, item selection strategy has a bigeffect on test efficiency, accuracy, safety, fairness and so on, which makes itemselection strategy very important in CAT.This paper adopts comparative research on Maximum Information method anda-Stratified method. The influence of different strategies on test results evaluatingmetrics was discussed in this paper, as well as the advantages and disadvantages ofthese two methods. Besides, this paper considers the relationship between testefficiency and item exposure rate in a-Stratified method. Thus this paper provides thereference for choosing appropriate item selection strategy in the process ofimplementing of CAT in future.This paper simulates a complete CAT process through computer simulationexperiment: firstly, generating examinee and item parameters through Monte Carlo method and checking item difficulty parameter and the ability of examinee to makesure they meet normal distribution requirements, and use these data to simulate itemsand examinee in the real world. Secondly, testing process—the simulation of responseof examinee to item and item selection process. In this process, choose the items withcorresponding difficulty level for each examinee according to his ability. Based onboth the item’s difficulty and the examinee’s ability the examinee’s answer to eachitem is simulated by using random probability. And according to the examinee’sanswer the algorithm updates the estimation value of examinee’s ability which waslater used in choosing the next item. In this phase two item selection strategies areadopted --- a-Stratified method and Maximum Information method. At last, severalmetrics were calculated to analyze the test results of each strategy, such as testefficiency, item exposure rate, test overlap rate, test accuracy and so on.Considering the principle of these two methods and the comparison ofexperimental results, this paper concludes that Maximum Information method onlyconsiders test efficiency without controlling item exposure rate, so experimentalresults show that it has higher test efficiency, but item exposure rate is extremelyuneven. However a-Stratified method makes some control on item exposure rate, thusa-Stratified method has more uniform item exposure rate, but it sacrifices a little testaccuracy. Experimental results indicate that usually increasing test efficiency andcontrolling item exposure rate conflict with each other, which means that these twoobjectives cannot be met at the same time, so there is a trade-off between them, whichshould be carefully weighed according to the specific situation in order to be betterused in CAT.