Test Fit Statistics Of Model-Data Fit Test In Cognitive Diagnosis Theory

With the development of psychometrics and cognitive psychology, cognitive diagnosis models have attracted more and more attention. Conventional test theory gives a direction to the scholars to analyze the participants‘ ability from a macro angle, but cognitive diagnosis theory(CDT) guides the researchers to explore the micro perspective, which emphasizes on the attributes or skills included in the ability and their structures. But the validity in all applications of CDT is dependent on the extent to which the selected model accurately reflects the data at hand. Only when the CD model fits the data well, can the advantages and functions of CDT emerge. Selection of the wrong model would lead to relatively large error in parameter estimation, test equating, the analysis of differential item functioning and so on, which would result in adverse effect. Therefore, it is required to evaluate model-data fit before applying CDT.Common fit statistics in the field of CDT consist of test fit statistics and item fit statistics; the former evaluate fit from test angle, the latter evaluate fit from item angle, which can guide the selection of items. Test fit statistics usually are relative fit statistics and item fit statistics usually are absolute fit statistics. In fact, it is very tedious and blind to use item fit statistic to assess model-data fit from one item to another, and absolute fit assessment is quite difficult. Thus, many researchers only consider test fit assessment, which can give a direction to model selection.Model-data fit in CDT is very important in the field of psychological and educational measurement and also easily neglected in test analysis process. It has not been found yet that there are any similar published articles. Test fit statistics can be expounded and compared from model misspecification and Q-matrix misspecification. Common test fit statistics in cognitive diagnosis consist of-2LL, AIC, AICc, BIC and DIC4. It is useful to the practical work to compare these statistics with CVLL, which performed well in item response theory. This article evaluated these statistics in different sample sizes, test lengths and numbers of attributes. Also, Q-matrix misspecification was considered. An empirical example involving real data was used to illustrate how the different fit statistics can be employed to identify misspecifications. This research proved that a) AIC and BIC performed better than AICc, b) CVLL performed best, c) AICc and BIC tended to select the reduced model than the saturated model in small sample size and long test length, d) when the number of attributes became 9, the correct rates of these statistics decreased, meanwhile, CVLL performed best, e) if the true model was R-RUM, the selection rates of correct Q-matrix of these statistics decreased except CVLL. The empirical example proved that DINA, NIDA and R-RUM neither fit the fraction data nor ECPE data adequately, but R-RUM is the best-fitting model among a set of competing models.