Identification of latent constructs and assessment of measurement equivalence across rating sources on a 360-degree performance appraisal instrument

When utilizing 360-degree feedback instruments it is taken for granted that direct comparisons between rater groups on the scales measured is appropriate. It is assumed that scale scores from each of the rater sources are directly comparable because all raters, regardless of their unique perspective of the target manager, are evaluating that person on the same underlying characteristics and on the same observed measurement scale. This assumption is referred to as measurement equivalence. However, it is possible that the underlying structure of the instruments may differ between rater sources, leading to the measurement of separate psychological constructs for each of the different rater sources. This would make direct comparisons between rater groups on the scales potentially inaccurate and misleading. This study examined the latent constructs that are being measured on a commercially available 360-degree performance appraisal instrument, and whether there is measurement equivalence between rater sources on these constructs. The sample consisted of 1,019 managers from a diversity of organizations and levels. The latent structure of the instrument was found to differ widely between the four rater sources (i.e., Self, Supervisor, Peer, and Direct Report). Three common factors, consisting of a proportionately small subset of items from the total instrument, were identified and confirmed using Confirmatory Factor Analysis (CFA). Measurement equivalence across rating sources was examined on each of the three factors using both Simultaneous Confirmatory Factor Analysis (SCFA) and an item response theory-based procedure, Differential Functioning of Items and Tests (DFIT). Using SCFA, two of the factors were found to lack measurement equivalence across rating sources. At the item level, proportionately more items were found to lack measurement equivalence using the SCFA procedure than the DFIT procedure.