A Comparison Of Methods Of Detecting Publication Bias In Meta-Analysis

Background and objectThe publication bias is an important factor of affecting the quality ofmeta-analysis. The funnel plot is widely used to detect publication bias though thereare several quantitative methods to do so. The different methods would make theresults difference because of their fundamental of theory. It is interesting that whichmethod is best to detect the publication bias. There were less literatures to beconcerned with this problem. The object of this study was to compare five methods ofdetecting publication bias and provide some instructional suggestion for the application.Methods:Five methods of bias detection, Egger's regression method, Begg's rankcorrelation method, funnel plot regression, trim and till, Richy's method, wereconsiderd in the study.We took four-fold table as data type in this research. Monte Carlo technique wasused to simulate meta-analysis data with or without publication bias which wasdetected by five methods. The scenarios were as follows.Simulations of meta-analysis data without publication.·Set the true effect size (log OR).·Set the study number in meta-analysis (k).·Set the sample size of studies in meta-analysis. It was supposed that the study numbers of two groups were equal, and the studies in meta-analysis came fromnormal or lognormal distribution.·The underlying outcome proportion in the control groupÏ€₀ was taken to havea uniform distribution between 0.1 and 0.5. Once it's determined by a numbergenerator, the corresponding proportion in the treatedÏ€₁ was calculated from theexpressionÏ€₁=or*Ï€₀/(1-Ï€₀+or*Ï€₀).·The simulated results for each study were generated using a binomial randomnumber generator. The process was repeated k times to make up a meta-analysis data.The meta-analysis with publication bias was generated from meta-analysis thatwithout publication bias. We excluded some studies with large p-value by the weightfunction w_i(p_i)=exp{-4p_i^1.5}.Results were evaluated in terms of typeâ… error control and statistical power. Theempirical typeâ… error rate for each test was ascertained for the scenario of nopublication bias. The empirical power was determined for situations wherepublication bias was present.The simulations were mainly performed by SPSS and the macro facility and theODS function in SAS.According to the pilot simulation study, the results trended to stability whensampling repeated 10000 times. Sampling was based on fixed random seed with10000 times for each procedure.ResultFor both the scenarios that k had a normal distribution and lognormaldistribution, the ranks of typeâ… error for five methods of detecting publication biaswas Richy's method, Egger's regression methods, Begg's rank correlation method,funnel plot regression, trim and fill method. Only funnel plot regression resulted intypeâ… error control that was close to the nominal alpha level 0.05. The ranks ofpower were Richy's method, Egger's regression methods, Begg's rank correlation method, trim and fill method, funnel plot regression. Besides Richy's method, themean power of the left four methods did not exceed 0.5.For both the scenarios k had a normal distribution and lognormal distribution,the typeâ… error and power of Egger's regression methods, Begg's rank correlationand Richy's method increased and decreased asμincreased, while funnel plotregression and trim and fill method had small variety.For both the scenarios that k had a normal distribution and lognormaldistribution, the use of Egger's regression, Begg's rank and Richy's method resultedin liberal typeâ… error control (with the estimated typeâ… error rates exceeding 0.90 insome conditions). The mean power of Egger's regression and Begg's rank exceed0.70 and 0.50 separately when the true effect size was little than -0.916(corresponding to odds ratios of 0.4). Conversely, the power of Richy's method had asmall variation.For both the scenarios that k had a normal distribution and lognormaldistribution, the mean power of Egger's regression was 0.10, and the mean power ofEgger's regression was 0.10. They increased rapidly when the true effect sizedecreased (with the estimate of power to 0.99). Conversely, the left three methods hada small variation when the true effect size decreased.For both the scenarios that k in meta-analysis had a normal distribution andlognormal distribution, the typeâ… error and power of all the five methods increasedwhen k increased. Only that the typeâ… error of trim and fill increased by 0.01, and0.16 of power.For both the scenarios that k had a normal distribution and lognormaldistribution, the typeâ… error and power of Egger's regression and Begg's rankdecreased when the variance of sample size of study in meta-analysis increased, whileRichy's method increased.The mean typeâ… error of Egger's regression and Begg's rank correlation whenthe sample size of study in meta-analysis had a lognormal distribution was small thanthat had a normal distribution. While other methods were larger that had a normaldistribution. It's so as the power. Discussion and ConclusionIn general, for Egger's regression and Begg's rank, they were all sensitive to thetrue effect size. When the true effect size was 0 (corresponding OR was 1), their meanpower were 0.10, which was close to their typeâ… error. And when the true effect sizewas -1.609 (corresponding OR was 0.2), their mean typeâ… error could exceed 0.90. Itfollowed the research of Petra and Jeffrey, though the parameter of true effect sizevaried small. It is possibly that the simulation in this study was not comprehensive.For the situation that the typeâ… error and power decreased when the variance ofthe sample size of study increased for the above two methods, it is possibly that theywere all based on sample effect size variance.Funnel plot regression resulted in typeâ… error control that was close to thenominal alpha level 0.05, but the power of it was very small.For the method of Egger's regression, Begg's rank correlation and funnel plotregression, the typeâ… error and power increased when k increased obviously, whichfollowed the results of Petra and Jeffrey. It is possibly that they are based onregression or correlation. So we hope that if the number of study is large (for examplebigger than 30), we do not choose these methods to detect publication bias.Trim and fill had a small typeâ… error (the mean of it was less than 0.01) and asmall power (the mean power was 0.3), as the results of Jeffrey showed. It is possiblythat k₀ is calculated by some extreme values.Richy's method was conservative to the other four methods, for typeâ… error wasrelatively large, except when k is equal to 20. It increased rapidly when k increased. Itis because its judgment interval is 95% nonparametric confidence interval.The simulation results showed that there were some defects in all fivequantitative methods to detect publication bias. But when k is small than 20, wesuggested to choose Richy's method to detect publication bias.We should choose several methods together to detect publication in the processof data analysis, and judge combined with sensitivity analysis and the real data, so asnot to make mistakes.