A Study On Statistical Methods Applied In Evaluation Of The "Global Minimum Essential Requirements In Medical Education"

BackgroundThe Institute for International Medical Education (IIME) formulated the Global Minimum Essential Requirements in Medical Education (GMER) In 2001, and administered a baseline evaluation in 8 leading medical schools of China to assess the competences of medical graduate students In 2003. In this evaluation, seven domains of GMER wereevaluated by using three measurement instruments-Multiple ChoiceQuestions (MCQ), Objective Structured Clinical Examination (OSCE) and faculty observation(OBS).ObjectivesBased on one administration criterion-referenced test data, the study aim to investigate reliability and validity of the measurement instruments (OSCE and OBS) and the overall GMER evaluation, as well as the cut-score dependability of each domain with each instrument, and to provide suggestions on how to improve GMER evaluation design and assessment quality.MethodsRandom one-facet crossed design, random two-facet crossed design and fixed model crossed design Generalizabiiity Theory (GT) were used to analyze the reliability and cut-score dependability of standardized patient (SP) stations in OSCE; Random two-facet crossed design, three-facet mixed design GT and a three-factor random two-facet crossed design Multivariate Generalizabiiity Theory (MGT) were employed to estimate the reliability and cut-score dependability of OBS; A traditional multitrait-multimethod (MTMM) matrix and a MTMM Confirmatory Factor Analysis (CFA) model were used to analyze the convergent and discriminate validity of the GMER evaluation; One- and two-order CFA were applied in examining the construct validity of OSCE; And the CFA were also used to analyze the construct validity of OBS.ResultsClinical graduates of seven-year program in Sichuan University reached the GMER standards in a whole. The reliabilities of SP stations in OSCE are unsatisfied, but the cut-score dependability of that is very high; also, the reliabilities and cut-score dependability of OBS are low. The construct validity of the overall evaluation is not very satisfied, both measurement instrument factor and trait (competence) factor greatly affect evaluation outcomes, in addition, the measurement instrument effect is stronger than trait (competence) effect; the construct validity of OSCE is acceptable and that of OBS is very satisfied.ConclusionsGT is applicable in assessing reliability and cut-score dependability of criterion- referenced test, which can not be well estimated in the frame of classical test theory (CTT). The reliability of SP stations in OSCE and that of OBS need to be improved. CFA is appropriate in analyzing the consistency of observed competences and expected measurement objectives. The construct validity of the overall GMER evaluation need to be improved and the measurement instrument effect should be reduced. Suggestions: ? In OSCE, Increasing long case SP stations to 10 and short case SP stations to 5, and setting similar cases could be expected to improve the reliability of the SP stations. ? In OBS, increasing internship phases to 6, observing in similar departments, and students being rated by the same observers in one department could improve the reliability of OBS; And the cut-scores of OBS in different domains need to be rebuilt. (§) In OBS, the items to evaluate "Communication Skills" need to be redesigned and the item 13 "Accepting other health professionals' roles" should be explained more clearly. ? More efforts to train raters and students' chances to learn about the measurement instruments are needed to reduce the effects of instruments.