An examination of effects on decision accuracy of changes in exam length, case selection, and scoring method in complex performance assessments

This study examined effects of various item (case) selection factors on the decision accuracy of complex medically-related performance assessments (standardized patient examinations [SPEs]) under conjunctive and compensatory scoring procedures. The conjunctive scoring procedure required passing scores on each of a case's three subskills (interpersonal, history-taking, and physical examination skills). The compensatory scoring model used a total score cutpoint for making the case-level mastery decision. Rasch model item difficulty estimates and examinee proficiency estimates were obtained at case and subskill-within-case levels based on empirical data from 441 medical students on a 13-case SPE. These real data formed the basis for a large-scale simulation study examining mastery decision accuracy under five sets of conditions that varied the number of cases and method of case selection: five and ten randomly selected cases, five maximally reliable cases, five most difficult cases, and all 20 cases (baseline). For the simulations, the number of cases was artificially increased to 20 by sampling additional item parameters. For each of the five study conditions, 1000 samples were simulated, each with 1000 simulated "examinees". Proficiency scores for "examinees" were generated to match the estimated sampling distributions of the 441 real examinees. The simulated examinees' pass/fail status was compared to their observed pass/fail status under both conjunctive and compensatory scoring.; Use of the five most reliable cases produced the most accurate results based on evaluation of kappa statistics and false-positive and false-negative decisions. Selection of the five most difficult cases was the least accurate method for this high-ability examinee group; the proportions of false negative outcomes were worse than randomly selecting cases. Decision accuracy generally increased as the number of cases on the exam increased. Under compensatory scoring, exams constructed from the five most reliable cases were nearly as accurate as those with ten randomly selected cases. Conjunctive scoring was demonstrated to be a very poor choice for scoring across all conditions.; The development of complex performance assessments requires many decisions, including the number of scorable elements, methods for selecting those elements, and scoring procedures. This study demonstrated the separate effect of each assessment characteristic on the accuracy of decisions made from such assessments, as well as the strong detrimental effect of a conjunctive scoring rule across all other conditions. Generalizations, limitations, and areas warranting further research are discussed.