| A multi-faceted model is developed to demonstrate the consequences of sample size choice in usability testing. This model takes into account the severity levels of user performance, the distribution of severity levels, the impact of usability problems, the benefit of improving user interface components, and the decision-making regarding which usability problems to solve. A large-scale usability test (N=103) of two products (Connexions and Snapfish) was carried out to provide real-world empirical data for the multi-faceted model. Simulations based on both the empirical data and other possible situations in usability testing were done to demonstrate the sample size influence on the estimation of severity, impact, and benefit given various types of parameters. The results showed that there were great risks in using small samples in various situations, especially when the shapes of the distributions of severity levels were positively skewed or bimodal. The results also indicated that user distributions at different severity levels and the impact weights of the usability problems had the most influence whereas the improvement factors and the improvement schemes had little influence on the sample size requirements. In addition, the user distribution data from the empirical study were transformed to dichotomous data. The analysis of the dichotomous data, together with the simulations based on the dichotomous data and the binomial model used frequently by the previous researchers, showed that the assumptions of usability problem independence and homogeneous users for the binomial model were seriously violated in such a way as to greatly overestimate the rate of problem detection. |
