Statistical consequences of attribute misspecification in the Rule Space Model

Psychometric models for cognitive diagnosis are designed to infer an examinee's abilities on certain attributes, meaning the specific skills, knowledge, and cognitive processes needed to solve particular test items correctly. In order to apply these cognitive diagnostic models, relevant attributes involved in the solution of specific test items must be identified and represented in an incidence matrix, called a Q-matrix. Correct attribute specification is a fundamental step and also predetermines the set of latent classes, termed knowledge states, into which students are classified. The present research examines some statistical consequences of attribute misspecification in the Rule Space model of Tatsuoka (1987, 1990). The current research examines statistical consequences of two types of attribute misspecification in the Rule Space Model. Study 1 examines the statistical consequences of using a misspecified list of attributes arising when either an essential attribute is excluded or a superfluous attribute is included. Study 2 investigates a case where different levels of mastery of an attribute are required to correctly solve particular items. Study 3 considers the effects of attribute exclusion in a Guttman-structured Q-matrix, in which attributes are totally ordered and test items measure a unidimensional ability. In these studies, data are simulated based on ideal response patterns identified by a Q-matrix with added error, so that the simulated responses are aligned with attribute specification in the Q-matrix. Results show that exclusion of an essential attribute generally results in underestimation of examinees' attribute mastery probabilities for the remaining attributes, and inclusion of a superfluous attribute leads to overestimation of examinees' true attribute mastery probabilities. Two factors systematically influencing the consequences across three studies are found to be the number of items that each attribute is involved in and inclusion relations among attributes.