Simulation study on the validity of methods for detecting publication bias in meta-analysis for binary outcomes

Due to sampling error and true heterogeneity, a single study cannot provide a comprehensive picture and a precise estimate of, say the effectiveness of a treatment. Systematic reviews that identify and integrate relevant studies have become the most important scientific, quantitative method to summarize scientific research. Meta-analysis is the statistical method used in systematic reviews to combine results from individual studies.;However, due to selective submission and publication, not all relevant studies conducted, especially those unpublished studies with an insignificant negative result, are easily accessible to those who conduct reviews. As a result, the truth, say, the effect of a treatment, would be overestimated. This phenomenon is known as publication bias. A few methods for detecting the bias have been developed and used in meta-analyses. Although their accuracy has been studied, some important issues remain to be answered, such as when would a method be good enough for practical use and is it similarly good for different definitions of the odds ratio?;Methods. We conducted a simulation study to examine the accuracy of four commonly used bias-detection methods with various ORs and P1-P2 combinations. In a simulation study, the true bias status can be predetermined and thus be compared with the results of the bias-detection methods. The four methods are Egger's regression, funnel plot regression, rank correlation regression, and Tang's regression. Realistic sample size was used for simulating individual studies and the numbers of studies in a meta-analysis was also varied. Both the sensitivity and specificity are examined against the magnitude of the OR and the P1-P 2 combination to identify the ORs and P1-P 2 combinations for which a method is sufficiently accurate. Predictive values are also examined for the same reason and in the same manner.;Results. The sensitivity and positive predictive value are generally low and in particular when the OR is close to one for which publication bias is of a particular concern. Egger's regression has the highest sensitivity among the four, in particular when the OR is neither close to one nor exceptionally large or small. Due to the relatively lower specificity, the positive predictive value of Egger's regression is not as high as that for Tang's regression and funnel plot regression. Tang's regression and funnel plot regression are very similar in sensitivity, specificity and predictive values, with the former being slightly better. Rank correlation seems the least accurate method overall. Tang's regression has in general the highest positive predictive value among the four methods in particular when the OR is below one.;Conclusions. The sensitivity and positive predictive value are generally more concerned than the specificity and negative predictive value in assessing and adjusting publication bias in meta-analyses. In this sense, Egger's regression can be recommended for its high sensitivity, while any positive result from Tang's method would suggest a probability of bias that should be taken seriously. Given the different patterns of the accuracy with the OR and the P1-P2 combination, a combination of Egger's regression and Tang's regression would be advisable. Further studies are needed to study the accuracy of methods used in combination.