| Meta-analysis has flourished in recent years as a popular method of research synthesis. Similarities which have not been rigorously investigated exist between meta-analysis and other problematic synthesis methods familiar to economists. Social choice decision-making as characterized in Arrow's General Possibility Theorem (1963) and index number calculation treated by Fisher's The Making of Index Numbers; a study of their varieties, tests, and reliability (1927) are both aggregation procedures that yield "aggregation paradoxes," defined by Saari (1994) as mathematically counter-intuitive conclusions from aggregation procedures. Using a comparison of meta-analytic techniques applied to fixed data sets, I demonstrate that meta-analysis is a similar aggregation procedure and can produce paradoxical results; for the same hypothesis test, different methods produce opposing conclusions.;This research utilizes the literature of education and medicine; both rely heavily on meta-analysis. Three topics are addressed. (1) I evaluate the extent to which meta-analysts agree about how to perform meta-analysis from the perspectives of both those who perform meta-analysis (practitioners) and those who are cited as meta-analytic experts by practitioners. Little substantive agreement about important meta-analytic characteristics is detected between and among experts and practitioners of meta-analysis. (2) I examine the results achieved by different meta-analytic techniques when applied to the same data set. A set of 52 meta-analyses are drawn from published medical and educational literature. Hypothesis tests identical to those reported are performed with test statistics calculated via different techniques. For twelve studies, the hypothesis tests agree in all reanalyses. The remaining 40 studies, however, display rates of ambiguity that range from 50 percent agreement to 95 percent. (3) I address the study- and technique-specific characteristics that may affect the outcome of a meta-analytic hypothesis test. Characteristics of data sets and of techniques prove to influence outcomes. However, how closely the meta-analyst conformed to industry guidelines did not influence outcomes. It is clear that the meta-analytic method selected matters critically to the outcome of a meta-analysis, therefore, policy decisions based on meta-analytic results need to be carefully reconsidered to ensure that the results on which the policy is based are robust. |
