Multilevel factor analysis by model segregation: Comparing the performance of maximum likelihood and robust test statistics

Survey measures of classroom climate and instructional practice have become central to policy efforts that assess school and teacher quality. Measures of classroom climate are often formed by aggregating individual survey responses. This has sparked a wide interest in using multilevel factor analysis to test hypotheses about the psychometric properties of classroom climate variables. One approach to multilevel factor analysis is conducted in two steps. First, the total covariance matrix is partitioned into separate between-group and within-group covariance matrices. Second, conventional factor analysis is used to test models separately. This study shows that when using this approach, rescaled and residual-based test statistics provide better inferences about the between group-level measurement structure than Maximum Likelihood test statistics even when the number of groups is large and there is no excess kurtosis in the observed variables. This study presents an empirical example and a simulation study to demonstrate how item intraclass correlations and within group sample sizes influence test statistic performance.