Examining the psychometric properties of a multimedia innovative item format: Comparison of innovative and non-innovative versions of a situational judgment test

In the measurement field, innovative item formats have shown promise for increasing the capability to assess constructs not easily measured with traditional item formats. These innovative items are often assumed to also provide opportunities for better measurement. However, little empirical research exists to support these assumptions. The purpose of this study was to explore the psychometric properties of a multimedia innovative item type and then compare the results to the properties of a non-innovative item format. Participants were administered one of two tests of identical content: one consisting of an innovative item format and the other consisting of a non-innovative item format. Exploratory factor analyses were conducted to evaluate the dimensionality of the two tests. The graded-response model was fit to both tests to produce item and test level characteristic curves, allowing for the examination of the reliability, or information, produced by each test and the individual items. Measurement efficiency, a ratio of the average amount of information provided relative to the average amount of time taken, was also reviewed. Face validity was examined by analyzing participant ratings on an eight-item post-test survey. Finally, criterion-related validity was investigated for the innovative item format by examining the relationship between test scores and supervisors' ratings of employee performance.;Findings from this research suggest that the use of innovative items may alter the underlying construct of an assessment, and could potentially provide more measurement information about examinees with low prioritization skills. Also, innovative item formats do not necessarily decrease measurement efficiency, as has been previously suggested. Participants' perceptions of the tests indicated that they felt the innovative version provided a more realistic experience and increased levels of engagement. Criterion-related validity estimates on the innovative version were inconsistent across two samples.;The key implication of these results applies to any practitioner employing innovative items: the addition of innovative item formats likely alters the measurement properties of a test. Further examination is needed to understand whether or not the alteration results in better measurement. As the overall psychometric functioning of both versions of the assessment was low, replication is recommended prior to generalizing these results.