Assessment of quality of studies for inclusion in metaanalyses

The purpose of this study was to develop and evaluate an approach to assessing the methodological quality of studies for inclusion in meta-analyses. A survey of meta-analyses in education identified three methods in use: coding the studies on multiple methodological variables, excluding studies judged not sound methodologically, or no mention of methodological quality. Without an assessment of the influence of quality, confidence in the outcome is jeopardized. Prior exclusion impoverishes findings.; A tool, the Methodological Quality Assessment Tool (MQAT) was developed and used to reanalyze four meta-analyses selected as representative of the three methods. Use of the MQAT results in two scores, a methodological quality score (MQ) and a reporting clarity index. For the MQ score, modified percent agreement for two raters on 20 studies was 85%. The correlation was.62. The distribution of scores was examined and studies divided into higher and lower quality studies. A cluster of studies was identified with highest scores for both methodological quality and reporting clarity. The mean effect size for these groupings was calculated using the original meta-groupings was calculated using the original meta-analyst's reported effect sizes for studies. In two out of the four reanalyses, the mean effect size of higher quality studies supported the meta-analyst's findings of a moderate effect. The cluster of best studies had an even larger effect. In the third reanalysis, the higher quality studies had a lower mean effect size and the cluster of best studies had an insignificant mean effect. This finding was contrary to the meta-analyst's finding of a moderate effect. The fourth meta-analysis reanalyzed was one that excluded studies on the basis of methodological quality. A list of excluded studies was derived and the sample for the reanalysis included both the original studies and the excluded studies. The mean effect of the higher quality studies was of moderate size contrary to the findings of the original meta-analysis. Exclusion of studies also reduced the breadth of information available from the meta-analysis. Careful assessment of methodological quality permits estimation of the influence of study quality without this loss of information.