| The Sequential Probability Ratio Test (SPRT) was used as a method to determine the number of subjects needed during summative evaluation of a website. Extant methodology for choosing the number of subjects during formative evaluation has often been based on a Poisson model. The goal of formative evaluation is to identify and correct website usability problems. However, during summative evaluation, the primary goal is to evaluate website effectiveness. Does the SPRT, when used to determine website effectiveness: (1) reduce the number of participants and tasks needed, (2) retain sufficient problem identification, and (3) improve cost effectiveness of usability evaluation?; Twenty-five undergraduate pre-education students participated in usability tests that employed a thinking aloud protocol. Each user attempted 20 randomly sampled tasks from a pool of 120 typical tasks for a large website. Three SPRT methods, the Poisson-based model, and a classical sampling approach were used in retrospective data analyses. Resultant identification of usability problems was also observed with these statistical techniques.; Results indicated that a website effectiveness decision could be made with the SPRT after testing 5 users chosen at random, with an average of 12 tasks per user. However, this number of users and tasks identified 25% of known usability problems. Likewise, the Poisson-based approach identified 35% of the problems with these same 5 users but with 20 tasks per user. For over 10 years, the prevailing rule of thumb has been that 4 to 5 subjects will identify the usability problems when evaluating websites. Results from this study indicated that more subjects may be needed for identifying 80--85% of website usability problems. Results also indicated that the SPRT is more cost-effective than the Poisson model if summative evaluation and not problem identification is the goal of usability testing. |
