Comparison of ability estimates using unidimensional and multidimensional scoring models for dichotomously scored items

Unidimensional scoring models have been used to score multidimensional tests because of the lack of availability of computer programs to conduct multidimensional item response model analyses. The purpose of this study was to assess the degree to which the classical test score model (CTM) and the 2-parameter unidimensional item response model with single calibration of all items (UIRM (T)) and separate calibration of the items within the subtest (UIRM (S)) were able to recover (a) the scores obtained by and (b) the classifications made using the multidimensional 2-parameter compensatory item response model (MIRM) when the dimensionality of the test was known to be two.;All scoring methods ranked examinees similarly. The recovery of the correlations between the examinees' subtest scores was complex across and within scoring method. The RMSD was small except when the subtest scores were used and the test structure was complex. The classification results revealed high rates of agreement based on the mean percentage between the MIRM and each of the CTM, UIRM (T), and UIRM (S). Taken together, the results suggest that the use of multidimensional, unidimensional, and number-right scoring will not lead to differences when total test scores derived from multidimensional tests with simple structure are reported. The results are equivocal with subtest scoring when the structure is complex.;Simulated data were generated where the same samples of examinees were used across the scoring methods for each condition. The factors considered were the correlation between examinees abilities on the two dimensions, the mean differences on the two dimensions, the factor complexity, and the type of score reported. The agreement between the scores yielded by the MIRM scoring model and each of the remaining three scoring models was assessed by (a) the differences between correlations between examinees' scores, (b) the differences between the correlations between the examinees' subtest scores within scoring method, (c) the root mean square difference (RMSD) between examinees' scores, and (d) the differences in rates of correct and incorrect classification of examinees.