Space-filling Exploratory Experimental Design

Monte Carlo simulations are widely used to study and compare statistical methodologies. Space-filling Exploratory Experimental Design (SEED) is a general approach to design extensive experiments to arrive at more general conclusions from a simulation study. SEED obtains performance measures of methods on a very large number of generative models that are systematically varied to ensure coverage in the space of models using mathematical optimization algorithms. Statistical modeling techniques are used to characterize how features of the underlying generative models affect performance. Furthermore through calibration, the fitted statistical model can be used to predict a method's performance on a new dataset. We present a simulation optimization algorithm to search for feature values where methodologies outperform or break down. The issue of reproducibility in research is addressed in SEED by using object oriented programming to design and implement the SEED framework. The modular code structure in SEED makes the simulation experiment easy to understand and modify; and all experimental details are systematically recorded to facilitate reproducibility.