Two Applications of Quantitative Methods in Education: Sampling Design Effects in Large-Scale Data and Causal Inference of Class-Size Effect

This dissertation is a collection of four papers in which the former two papers address the issues of external validity concerning incorporating complex sampling design in model analysis in large-scale data and the latter two papers address issues of internal validity involving statistical methods that facilitate causal inference of class size effects.;Chapter 1 addressed whether, when and how to apply complex sampling weights via empirical, simulation and software investigations in the context of large-scale educational data focusing on fixed effects. The empirical evidences reveal that unweighted estimates agree with the weighted cases and two scaling methods make no difference. The possible difference between weighted single versus multi-level model may lie in the scaling procedure in the latter. The simulation results indicate that relative bias of the estimates in the models of unweighted single level, unweighted multilevel, weighted single level and weighted multi-level varies across different variables, but unweighted multilevel has the smallest root mean square errors consistently while weighted single model has the largest values for level-one variables. The software finding indicates that STATA and Mplus are more flexible and capable especially for weighted multi-level models where scaling is required. Chapter 2 investigated how to account for informative design arising from unequal probability of selection in multilevel modeling with a focus of the multilevel pseudo maximum likelihood (MPML) and the sample distribution approach (SDA). The Monte Carlo simulation evaluated the performance of MPML considering sampling weights and scaling. The results indicate that unscaled estimates have substantial positive bias for estimating cluster- and individual-level variations, thus the scaling procedure is essential. The SDA is conducted using empirical data, and the results are similar to the unweighted case which seems that the sampling design is not that informative or SDA is not working well in practice.;Chapter 3 examined the long-term and causal inferences of class size effects on reading and mathematics achievement as well as on non-cognitive outcomes in early grades via applying individual fixed effects models and propensity scores methods on the data of ECLS-K 2011. Results indicate that attending smaller class improves reading and math achievement. In general, evidence of class size effects on non-cognitive outcomes is not significant. Considering potential measurement errors involved in non-cognitive variables, evidence of class size effects on non-cognitive domain is less reliable. Chapter 4 applied instrumental variables (IV) methods and regression discontinuity designs (RDD) on TIMSS data in 2003, 2007 and 2011 to investigate whether class size has effects on eighth grader's cognitive achievement and non-cognitive outcomes in math and four science subjects across four European countries (i.e., Hungary, Lithuania, Romania and Slovenia). The results of the IV analyses indicate that in Romania smaller class size has significant positive effects on academic scores for math, physics, chemistry and earth science as well as for math enjoyment in 2003. In Lithuania, class size effects on non-cognitive skills are not consistent between IV and RDD analyses in 2007. Overall, the small class size benefit on achievement scores is only observed in Romania in 2003 while evidence of class-size effects on non-cognitive skills may lack of reliability.