Evaluating Person-fit For Cognitive Diagnostic Tests With Polytomous Items

Cognitive diagnostic assessment(CDA)has been a hot topic of research in the measurement community for the past few decades.The advantages of CDA are currently receiving a great deal of attention from researchers in the areas of educational assessment,psychological assessment and situational judgement testing,particularly in educational measurement.The main purpose of educational and psychological measurement is to obtain information about an examinee’s potential traits,such as knowledge and skills in a subject area,individual attitudes and emotions.However,in practice,examinees are influenced by a range of additional factors that threaten the validity of the test results.These factors often lead to systematic errors in the measurement data,which are also referred to as abnormal responses.When assessing abnormal responding behavior,researchers often use person-fit statistic(PFS)to identify any abnormal patterns.Person-fit statistics studies is one of the key areas of research in cognitive diagnostic theory.However,few studies have explored person-fit statistics within the framework of the polytomous cognitive diagnostic model(CDM).Therefore,this study compares the performance of the PFS l_z,outfit and infit and their standardized versions(outfit_z and infit_z)in a polytomous CDM through two simulation studies and one empirical study to provide practitioners with some reference and insight when selecting individual fit statistics for multilevel scoring CDM.Study 1 extended and transformed the formulae for the l_z,outfit,infit,outfit_z and infit_z statistics in polytomous CDM framework to determine the distribution characteristics of the five PFS and their critical values under different experimental conditions.To examine the Type I errors rates and detection power of the l_z,outfit,infit,outfit_zand infit_z statistics under different experimental conditions and to compare them with each other,and to explore the effects of item quality,number of examinees,proportion of misfit and number of items on the five PFSs.Study 3 uses data from the PISA(The Program for International Student Assessment)science data to conduct an applied study of empirical data to further illustrate the performance of the five PFSs.The results of the study show that(1)The l_z,outfit_z and infit_z statistics are close to standard normal distributions,while the outfit and infit statistics are positively skewed.The distribution pattern of the fitted statistics is affected by the quality of the items,and the length of the test.The data suggest that the infit and outfit statistics do not apply to fixed critical values.(2)The l_z,infit and infit_z statistics are better at controlling for Type I errors rates under a variety of conditions,while the outfit_z and outfit statistics have weaker control over Type I errors rates.In terms of detection power,the l_z statistic has good power to detect all three types of anomalies,the infit and infit_z statistics have more stable power to detect creative responses,even better than the l_z statistic,and the outfit_z and outfit statistics need to be under more ideal conditions to have acceptable statistical test power,with all five PFSs being affected by item quality,the proportion of people misfit and the number of items.(3)In the empirical data,the detection rate of abnormal examinees was about 4%,and a random selection of abnormal subjects were analyzed.It was found that the five statistics had a high detection rate for the two types of anomalies,high scores for low ability and low scores for high ability.