Assessment of communication, professional and surgical skills in an Objective Structured Performance Related Examination (OSPRE): A psychometric study

Problem statement. The CanMEDs competencies for surgical residents are not well assessed. The primary research problem guiding the present study concerns the reliability and validity of the OSPRE methodology for simultaneously assessing communication, professionalism and surgical skills.;Method. Performance of communication, professional and surgical skills was rated using checklists and global rating scales in an OSPRE format. Fourteen residents from the General Surgery program of the University of Calgary were assessed in seven surgical simulation stations utilizing a combination of OSATS and MOMS models modified by simultaneously embedding the assessment of communication and professionalism skills.;Results. The reliability of the OSPRE for assessing competencies using checklists and global rating scales combined was: communication (alpha = 0.75 - 0.92), professionalism (alpha = 0) and surgical skills (alpha = 0.86 - 0.96). The OSPRE had construct validity for assessing surgical skills as a function of PGY level. For assessing communication there was no statistical significant difference between PGY levels. The modified professionalism checklist used in the OSPRE could not be validated in this setting. Surgical skills and communication had a good correlation in the two stations tested (r = 0.55 - 0.57).;Conclusions. The OSPRE methodology is a valid and reliable tool for simultaneously assessing the competencies of surgical skills and communication skills. However, in order to assess professionalism appropriate instruments need to be developed.