Reliability, validity and sources of errors in assessing physician performance in an objective structured clinical examination: A generalizability theory analysis

Background. International Medical Graduates (IMGs) from British Columbia, Alberta, Saskatchewan, Manitoba, and the Yukon participated in a high-stakes, fourteen case Objective Structured Clinical Examination (OSCE). The OSCE was part of the clinical competency assessment process developed for a demonstration project funded by Health Canada to incorporate qualified IMGs into the Western Canadian physician workforce.;Methods. Thirty-nine candidates rotated through fourteen 12-minute cases. Ten minutes were assigned for the candidate/standardized patient (SP) interaction, which was followed by a two-minute post-encounter probe. Each case was evaluated using a case-specific checklist, a five point global rating of overall performance, and a communication skills checklist. One presiding physician examiner completed all three of the assessment instruments, while the SP only completed the communication skills checklist. For the physician versus non-physician study, the physician rater trained two non-physician raters on the necessary assessment protocols for each case. Fifteen candidates were randomly selected and their videotaped interactions with the SP were evaluated using the case-specific checklist and communication skills checklist.;Results. The reliability coefficients ranged from low to moderate on the assessment formats. Generalizability analyses revealed large variance components attributed to SPs and examiners which suggest the candidates' scores were being unduly influence by physician examiners and SPs. The communication skills assessment revealed moderate to high internal consistency for each case. Low to moderate internal consistency was calculated for each checklist item across cases (inter-item). The percent variance for each checklist on participants, cases, p x c, and for the nested SPs and assigned physician examiners varies considerably from item to item, suggesting are differences in how the items are being rated based on the requirements of the case.;Physician raters and non-physician raters showed statistically significant differences on the checklist score in four cases, differences in the global scores on three cases, and differences in the total score on three cases. The intra-class coefficient ranged low to high on the physician examiner data and moderate to high on the non-physician examination data.;Purpose. The primary purpose of the present study was to identify the sources of error affecting the reliability of assessments of physician competence. The secondary purpose was to evaluate the reliability of physician versus non-physician examiners to score performance-based assessments from videotape recordings of OSCE cases. Generalizability analyses and other statistical procedures were utilized to study the psychometric properties of an OSCE developed to evaluate clinical performance in several domains such as history taking, application of physical examination skills, and the ability to communicate and/or counsel patients.;Conclusions. Low reliability results and large variance components attributed to physician examiners and SPs call into question the reliability of the OSCE for high-stakes assessment. Standardized training by the videotape physician rater aided in higher reliability coefficients with the checklist scores but the coefficients were lower on the global scores.