Comparison of cognitively diagnostic adaptive testing algorithms

Variations of two cognitively diagnostic adaptive testing algorithms were compared using a computer simulation based on full-test data consisting of 536 junior high school students' dichotomous responses to a 40-item fraction subtraction test. The two models forming the basis for the computer-adaptive testing algorithms were the Rule Space model developed by Kikumi Tatsuoka and her associates, and the Poset model developed by Curtis Tatsuoka and his associates. The computer adaptive testing (CAT) algorithms based on these models differ in terms of their item selection and test stopping criteria, and they were investigated for their efficiency and consistency in classifying individuals into their knowledge states, which are defined by the sets of mastered and non-mastered cognitive skills called "attributes." In this study, for CAT based on the Rule Space model, three different item selection methods were developed and compared for classification consistency, and two different test stopping methods, combined with the three item selection methods, were also compared for classification efficiency. Best-k-item tests based on the maximum R-squared were also evaluated against the CAT results. For CAT based on the Poset model, classification consistency and efficiency were first examined by evaluating the results of CAT and full-length testing. The two models were then compared to each other as well. CAT based on the Rule Space model seemed to work relatively well when a next item was selected based on an examinee's provisional knowledge state estimated from the examinee's responses made so far, and if the testing was terminated when no more items could drastically change the examinee's position in rule space, which is a space defined by IRT-theta and zeta (unusualness of responses). CAT based on the Poset model was very consistent and efficient in classification. It turned out to be more consistent and efficient than CAT based on the Rule Space model; nevertheless, both algorithms seemed to provide similar diagnostic classification of examinees as consistently as the full-length testing does.