Dual Objective Item Selection Strategies For Cognitive Diagnostic Computerized Adaptive Testing

In October 2020,the State Council of the Central Committee of the Communist Party of China Central Committee released the General Plan for Deepening Education Evaluation Reform in the New Era.It calls for improving summative assessments,strengthening formative assessments,making full use of information technology,and improving the scientific,professional,and objective nature of educational assessments.The advantage of Computerized Adaptive Testing(CAT)is that the test items are managed based on an algorithm,and the selected items are adapted to the latent traits of the students estimated during the test and then the items are circularly pushed in this mode.CAT is closely related to the measurement theories used,with item response theory being used for summative assessment to measure the latent traits(ability level)of the subject and cognitive diagnostic theory being used for formative assessment to measure the knowledge state of the subject.Both are irreplaceable as they reflect different aspects of the subject’s information.The test focuses on both latent ability and knowledge states,which can reduce administrations of multiple tests.The advantage of the Cognitive Diagnostic CAT(CD-CAT)is that it is possible to obtain information on both the latent traits(ability level)and the cognitive structure(knowledge state)of an examinee in a single test.The CD-CAT is a combination of educational measurement and computer technology,as well as a combination of summative and formative assessments.The Dual Objective CD-CAT significantly reduces the test length compared to a separate test to achieve the accuracy of both measurement objectives.Current research on the dual-objective CD-CAT has focused on the combination of item selection strategies,most of which favor the measurement of knowledge state,resulting in an imbalance in the accuracy of both ability and knowledge state information measures.Formative and summative assessments are of equal importance,and balancing two types of information is more important for dual-objective CD-CAT to than improving the accuracy estimate for only one type of information.Because of the close relationship between ability and knowledge state,also becomes important to use the information about the relationship between the two in the CD-CAT to improve the efficiency of the test.In view of the above application context and problems,we proposes a square weighted Kullback-Leibler(KL)optimization design for dual-objective CD-CAT selection that can balance the accuracy of ability and knowledge state measurement.The relationship between examinees’ latent ability in item response theory and knowledge state in cognitive diagnostic theory is explored,and its important value in the test is used to improve the item selection strategies to achieve efficient measurement.Based on the link between the latent traits,A new method of incorporating the joint distribution of ability and knowledge state as a priori information into the item selection strategy is proposed.The joint distribution of ability and knowledge state has also been extended as a priori information to the estimation of knowledge state,which has been used to improve the combination strategy of KL optimal design and some better performing dual-objective CD-CAT item selection strategies.The simulation and empirical results show that:(1)The square weighted KL-optimal combination strategy can balance the measurement accuracy of both ability and knowledge state information while balancing the efficiency of item selection and measurement accuracy;(2)The selection strategy with a priori information can achieve the same measurement accuracy with fewer test items,especially in the long test condition where the integration of a priori information improves the efficiency of question selection significantly;(3)Under the condition of the long test,the joint distribution of ability and knowledge state as a priori information can be extended to the item selection strategy and the estimation of knowledge state with better results.