An Improved Nonparametric Item Selection Method Of Cognitive Diagnostic Computerized Adaptive Testing For A Small-scale Education Setting

Cognitive diagnostic computerized adaptive testing(CD-CAT)combines the cognitive diagnosis assessment(CDA)method with computerized adaptive testing(CAT)to assess examinees’ knowledge structures,processing skills,or cognitive processes.The efficiency of CD-CAT depends on an effective item selection method.Therefore,administering the most ’appropriate’ items to the examinee is crucial for the test’s efficiency.The item selection methods in CD-CAT can be divided into parametric methods and nonparametric methods.Among them,parametric methods rely on the estimated item bank with parameters from a large sample.In nonparametric methods,nonparametric item selection method(NPS)is only suitable for simplifying models,while the general nonparametric item selection method(GNPS)is suitable for complex saturated models but also requires parameter estimation.To address these issues,an improved nonparametric item selection method(MNPS)is proposed in this paper based on the findings of Madison and Bradshaw.The MNPS method selects test items with fewer attributes while retaining the advantage that the NPS method does not require pre-calibration.Thus,it has excellent performance in data fitted by saturated models.Study 1 evaluates the advantages and disadvantages of Q-matrix design by comparing the influence of different number of attribute(s)measured by test items on the accuracy of nonparametric classification in the Q-matrix design.The accuracy rate of examinee’s attribute profiles classification is evaluated under different conditions,such as cognitive diagnosis model,Q-matrix design,attribute profiles’ distribution,item quality,and the method of estimation for attribute profiles.The results of study 1 show that the fewer the number of attributes investigated in the Q-matrix design,the higher the classification accuracy rate of the examinee’s attribute profiles under nonparametric classification method.Study 2 proposes an improved non-parametric item selection method(MNPS)that has excellent performance in fitting the item bank of the saturation model while not requiring pre-calibration.To examine the effect of the MNPS method,a simulation study was performed under multiple sets of conditions.Simulation results show that the MNPS method outperforms the GNPS,NPS,and other parametric methods when the calibration sample size is small,but the exposure rate of the item bank is high.The impact of Q-matrix design on the classification accuracy of non-parametric cognitive diagnosis methods and the MNPS method were considered firstly in this paper.However,how to balance the exposure rate and the accuracy of the item selection is the problem that the new method should face.MNPS may be suitable for the diagnosis of students’ attribute profiles at the beginning of learning,and the empirical research deserves further study.