Extensions to the Generalizability of Experiments in the Social Sciences

The following thesis is a collection of three research studies that serve as extensions to the field of causal generalization of experimental results in the social sciences. Over the past six years, policymakers have been increasingly interested in the extent to which experimental results hold in a larger population of inference. However, a common challenge to causal generalization, particularly in education, is that while treatment is randomly assigned to units, the experimental sample is not randomly selected from the target populations of inference. Recently developed methods to improve generalizations primarily use propensity scores to re-weight and match the experimental samples to the populations of inference. However, the effectiveness of these methods is challenged by the credibility and validation of assumptions in the model based frameworks and the complications that arise from small sample sizes. This dissertation considers different aspects of the causal generalization problem and presents alternative approaches to addressing the specific challenges of assumption credibility and small sample sizes in propensity score based methods.;Chapter 1 begins with an introduction to the problem of causal generalization in social science research and outlines in further detail the challenges to which the methods in this thesis seek to address. Chapter 2 extends partial identification methods to causal generalization with nonrandom samples by deriving interval estimates, rather than point estimates, of population average treatment effects. Partial identification methods consider the inferences that can be made when few to no assumptions are imposed on the empirical evidence. Because causal generalization problems typically incorporate population data frames, we consider the role of this auxiliary data source in contributing identifying power to tighten interval estimates. These methods are applied to a completed cluster randomized trial in education.;Chapter 3 shifts the focus from retrospective methods to causal generalization to prospective approaches by analyzing a meta-analytic approach to power computations with stratified sampling. In this chapter, we consider a new approach to power analyses with stratified samples by considering the strata as independent studies in a meta-analysis. We compare the power estimates under this meta-analytic framework with the traditional framework in cluster randomized and randomized block designs using empirical estimates of intraclass correlation coefficients in educational research. We consider the general case where the design is not necessarily balanced in both comparisons.;Chapter 4 returns to the retrospective approach to improving generalizations by extending applications of small area estimation to the problem of small sample sizes in subclassification methods. Small area methods were originally developed in the survey sampling field to improve the precision of estimators when limited by small sample sizes. An analogous problem occurs in propensity score subclassification where small experimental studies limit the number of propensity score strata that can be constructed, thus also limiting the bias reducing advantage of the subclassification estimator for causal generalization. We extend applications of small area estimation to this sparse propensity score strata problem by assessing the performance of small area estimators to estimate stratum specific outcomes. We compare the performance of small area estimators and direct estimators and assess the small sample properties of the former in a design-based simulation study using data from a completed cluster randomized trial. Chapter 5 concludes with final thoughts from these three studies and directions for future research.