The evaluation of multiple stage adaptive test designs

Traditional paper and pencil (P&P) tests are linear tests and the same questions are asked of every examinee during an administration. In computer adaptive testing (CAT), examinees receive items that match their current ability estimate. The last decade has seen paper-and-pencil (P&P) tests being replaced by computerized adaptive tests (CATS) within many testing programs. A multi-stage computer adaptive test (MST) combines characteristics of both a standard CAT and P&P test because it adapts to the ability of the examinee like CAT and provides P&P benefits such as test specialist review, exposure of pre-selected items and parallel test forms.; This dissertation evaluated several MST designs under the criteria of (a) scoring reliability to provide accurate scores, (b) simplicity of the design to facilitate review, and (c) efficiency of item pool usage to reduce the cost of test development. The dissertation presented models for constrained MST assembly and gives a computational study on solving the resulting problems. MST assembly models for both discrete and set-based question types have been considered. The dissertation used a commercially available code, CPLEX (ILOG, 2002), as the tool to obtain solutions to mixed integer programming problems arising from the assembly models. The mixed integer programming assembly models were written in AMPL, a modeling language for mathematical programming. Analytical techniques were developed to evaluate scoring accuracy using Item Response Theory (IRT).; The MST approach is being considered as a possible CAT implementation of the Law School Admission Test (LSAT). The final decision on the design choice rests with management and will be based on the objectives of the organization. Summary results necessary for management to make their decisions have been tabulated.