| BackgroundMeta-analysis is the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.The main purpose of meta-analysis is to estimate the overall effect size, including point estimation and interval estimation. The overall effect size is nothing more than the weighted mean of the individual effects. However, the mechanism used to assign the weights depends on our assumptions about the distribution of effect sizes from which the studies were sampled. Under the fixed-effect model, we assume that all studies in the analysis share the same true effect size, and that all differences in observed effects are due to sampling error. By contrast, under the random-effects model, we assume that the true effect size varies from study to study. For example, the effect size might be higher (or lower) in studies where the participants are older, or more educated, or healthier than in others, and so on. It is essential to quantify the amount of heterogeneity by estimating between-studies variance in meta-analytical inference. The between-studies variance represents the variability between the estimated effect sizes due not to within-study sampling error but to true heterogeneity among the studies, also called random effects variance or heterogeneity variance.The heterogeneity variance estimatorsSeveral methods of estimating the heterogeneity variance have been introduced. The most commonly used method in random effects meta-analysis was proposed by DerSimonian and Laird (DL) because it is simple and non-iterative. Two iterative estimators have been used in meta-analysis are based on maximum likelihood (ML) and restricted maximum likelihood (REML) estimation. The REML estimator is different from the ML estimator because the former is adjusted for the loss of degrees of freedom due to estimating the overall effect size and heterogeneity variance from the same data.Rukhin investigated the properties of the estimator proposed by Paule and Mandel (PM), showed that the PM estimator can be interpreted as a simplified version of the restricted maximum likelihood estimator, and also demonstrated that the PM method is a generalized Bayes procedure. Cochran introduced an estimator based on analysis of variance (ANOVA) of the data (CA), assuming that each study provides equal information. Kacker and DerSimonian found that three leading estimators (CA, PM and DL) are special cases of a general method-of-moments estimator, and they suggested two new two-step estimators (CA2and DL2). Sidik and Jonkman (SJ) proposed another simple and noniterative estimator by reparameterizing the total variance of an effect statistic, and an improved estimator by using a better priori value based on the CA estimator (SJ_CA). In addition, there are some other approaches for estimating the heterogeneity variance, including those of Hunter and Schmidt (HS), Hartung and Makambi (HM), and an empirical Bayes (EB) estimator.Among these twelve heterogeneity variance estimators mentioned above, PM, ML, REML and EB estimators require some computational programming to obtain an iterative solution, others are non-iterative estimators. HM, SJ and SJ_CA are non-negative estimators, others have to be truncated to zero when the estimators yield to be a negative value. CA, DL, PM and two two-step estimators (CA2, DL2) are special cases of a general method-of-moments estimate of the heterogeneity variance. At present, the commonly used meta-analysis softwares, including Revman, Stata, Comprehensive Meta-Analysis and Meta-Analyst, only contain DL, ML and REML methods for estimating the heterogeneity variance.There are a few published papers which have made comparisons of heterogeneity variance estimators. Sidik and Jonkman conducted a quite comprehensive Monte Carlo comparison of seven heterogeneity variance estimators (CA, DL, SJ, SJ_CA, ML, REML and EB), showed that the REML and especially the ML and DL estimators are not accurate, having large biases unless the true heterogeneity variance is small. However, the study is limited in the range of parameter values that the smallest number of studies included in meta-analysis is10. In practice, less than ten studies is a more common case.Confidence intervals for the overall effect sizeTo estimate the overall effect size from a set of individual studies, an average of the effect estimates is calculated by weighting each one of them by its inverse variance, and a confidence interval (CI) is thus obtained around it.The typical procedure to calculate a CI around an overall effect size assuming a standard normal distribution was proposed by DerSimonian and Laird, and it is the unique method including in commonly used meta-analysis softwares. Follmann and Proschan suggested using a t distribution, instead of the standard normal distribution, to produces CIs that are wider than those of the standard normal distribution, in particular for meta-analyses with a small number of studies, and consequently this should improve the coverage probability of confidence interval.Hartung proposed another method based on t distribution with a weighted extension of the usual variance, which keeps the prescribed significance level much better than the commonly used normal distribution test. Besides, a robust variance estimator could be used to calculate the CIs based on studen t distribution, given by Sidik and Jonkman. Brockwell and Gordon proposed another method by considering in particular the approximate0.025and0.975quantiles (QA) to calculate a95percent CI for the overall effect, using a Monte Carlo simulation. In addition, other methods of obtaining CIs that are computationally more complex are those of Biggerstaff and Tweedie (BT) and the profile likelihood (PL) method of Hardy and Thompson.Several studies have compared the coverage probabilities of CIs based on the standard normal distribution with one or two alternative methods. A comprehensive Monte Carlo study by Sanchez-Meca showed that the weighted variance CI outperformed the other methods regardless of the heterogeneity estimator, the value of heterogeneity, the number of studies, and the sample size. However, the study is oriented toward the effect size of standardized mean difference, and the method proposed by Biggerstaff and Tweedie and the profile likelihood method, were not considered, as well as the CIs with the robust variance estimator mentioned above.In summary, so far there is no certain answer which heterogeneity variance estimator is most accurate, especially when the number of studies is small. And the performance of different CIs for the overall effect size is still needed to be studied in the case that the odds ratio is selected as an effect size.ObjectiveIn this study, we conducted a comprehensive Monte Carlo simulation to compare the properties of different heterogeneity variance estimators and confidence intervals for the overall effect size by selecting parameter values more suitable for practical application.Monte Carlo simulationWe chosed odds ratio as the measure of effect size, on the grounds of its popularity in meta-analysis. Odds ratio will be transformed to log scale before calculating the overall effect by using the inverse of variance, so actually we used log odds ratio as the effect size.The main procedure of simulation is to generate data for k2×2tables, then to calculate sample log odds ratios and standard errors to compute different heterogeneity variance estimator and confidence intervals for the overall effect size. If any zero cells were generated, we added0.5to each cell for that study. For all methods for estimating the CI, a95percent confidence interval was calculated.In this simulation, three parameters were considered, namely overall effect size (0), number of studies (k) and heterogeneity variance (τ2). 1) θ=-1,-0.5,0,0.5,1, total5levels.2) k=3,5,8,10,15,20,30,50, total8levels.3) τ2=0,0.1,0.2,0.3,0.4,0.5,0.75,1,1.25,1.5,1.75, total11levels.Besides, the optimal weights, defined as the weight calculated using the heterogeneity variance value selected, were added to compute for comparison purpose when calculating the confidence intervals. Note that we selected the case τ2=0, when there is no heterogeneity, to investigate which CIs is best for fixed-effect model with the optimal weight, as well as for random-effects model with different heterogeneity variance estimators.We compared the accuracy of different heterogeneity variance estimators in terms of bias and mean squared error (MSE), and the coverage probabilities of different CI methods with different heterogeneity variance estimators, along with the coverage probability of the confidence interval based on standard normal distribution with the optimal weight.ResultsThe heterogeneity variance estimatorsPM and EB estimators appear to be the best among all the estimators, followed by SJ_CA estimator. The CA estimator in general overestimates τ2for all cases, and SJ estimator overestimates τ2for most cases. On the other hand, DL, ML and REML estimators underestimate the heterogeneity variance for most cases. HS and HM estimators generally underestimate τ2, so do CA2and DL2estimators. As expected, the difference in bias between ML and REML for a given value of τ2is decreasing as kincreases.The overall effect sizeThe estimated overall effect sizes based on each estimator are very similar among themselves for all cases. All estimators overestimate the overall effect with a small magnitude of bias, except when there is no heterogeneity.Confidence intervals for the overall effect sizeThe coverage probability based on the standard normal distribution with the optimal weight was very close to the nominal confidence level of95%(mean coverage probability=95.48%), as well as those obtained with the CIs based on weighted variance and robust variance (mean coverage probability=95.25%and94.94%, respectively). However, the CI calculated based on t distribution overstated the nominal confidence level (mean coverage probability=97.78%), as did the CI obtained by the QA method (mean coverage probability=98.08%).CIs based on the standard normal distribution presented coverage probabilities clearly under the nominal confidence level, the mean coverage probabilities ranged from89.09%(HS estimator) to93.27%(SJ estimator). With a given heterogeneity variance estimator, coverage probabilities were much variable in different cases among the twelve estimators. Forτ2=0, the mean coverage probabilities of CIs with different τ2estimators, range from95.88%(DL2estimator) to98.57%(SJ estimator), are some higher than the nominal level except those with the optimal weight (mean coverage probability=95.30%).CIs based on t distribution have a higher coverage probability than CIs based on z distribution. Almost mean coverage probabilities are close to the nominal level except the SJ estimator (96.76%). Forτ2=0, the mean coverage probabilities of CIs with different τ2estimators, range from97.94%(DL2estimator) to99.49%(SJ estimator), are some higher than the nominal level, so are those with the optimal weight (mean coverage probability=97.61%). Besides, all mean coverage probabilities reached100%when there is no heterogeneity and the number of studies is smallest (k=3, τ2=0), as well as the CIs with the optimal weight.The mean coverage probabilities were quite close to the nominal level when calculating CIs based on weighted variance for all cases. Furthermore, the variability in the coverage probabilities of the weighted variance CIs through the twelve estimators was quite small.As well as the CIs based on weighted variance, the mean coverage probabilities were quite close to the nominal level for all cases. Also, the variability in the coverage probabilities of the weighted variance CIs through the twelve estimators was clearly small.The mean coverage probabilities were also close to the nominal level when calculating CIs using the QA method. We would obtain coverage probabilities quite close to100%with the optimal weight when the number of studies is very small (k=3), and the coverage probabilities decreases as the number of studies increases. Forτ2=0, the mean coverage probabilities of CIs with different τ2estimators, range from98.22%(DL2estimator) to99.59%(SJ estimator), are clearly higher than the nominal level, so are those with the optimal weight (mean coverage probability=97.95%).The coverage probabilities of both BT and PL method were clearly under the nominal confidence level (mean coverage probability=91.29%and92.86%, respectively), and would be close to95%as the number of studies increases. Forτ2=0, the mean coverage probabilities of CIs are some higher than the nominal level,96.33%and96.76%for BT and PL method respectively. In general, the performance of PL is better than BT method.ConclusionsThe DL, ML and REML estimators will underestimate the true heterogeneity for most cases. PM and EB estimator is reasonably accurate in general. There is little difference between the overall effect sizes estimated by different heterogeneity variance estimators.The coverage probability of CIs based on the standard normal distribution was clearly under the nominal confidence level, especially when the number of studies is small (k≤10). So we suggest to calculate the CIs for the overall effect size based on weighted variance or robust variance, which perform better than other CI procedures, and can be robust for the case when the number of studies is less than10. Furthermore, they are also suitable for fixed-effect model. |
PDF Full Text Request