Quantifying the risk of error when interpreting funnel plots
 Mark Simmonds^{1}Email author
DOI: 10.1186/s1364301500048
© Simmonds. 2015
Received: 12 December 2014
Accepted: 21 January 2015
Published: 11 March 2015
Abstract
Background
Funnel plots are widely used to investigate possible publication bias in metaanalyses. There has, however, been little formal assessment of whether a visual inspection of a funnel plot is sufficient to identify publication bias.
Methods
Visual assessment of bias in a funnel plot is quantified using two new statistics: the Imbalance and the Asymmetry Distance, both intended to replicate how a funnel plot is typically assessed. A simulation study was performed to assess the performance of these two statistics for identifying publication bias.
Results
The two statistics both have high type I error and low statistical power, unless the number of studies in the metaanalysis is very large. These results suggest that visual inspection of a funnel plot is unlikely to lead to a valid assessment of publication bias.
Conclusions
In most systematic reviews, visual inspection of a funnel plot may give a misleading impression of the presence or absence of publication bias. Formal statistical tests for bias should generally be preferred.
Keywords
Publication bias Funnel plots Metaanalysis Egger’s testBackground
Publication bias arises where studies with results that go against the prior opinion of the authors (such as when a new drug is less effective than placebo) or are not statistically significant are not published, and so are not included in a systematic review and metaanalysis, leading to biassed conclusions [1]. It is therefore important to assess whether publication bias might affect a metaanalysis. A commonly used method to assess whether this is the case is to examine a funnel plot. This is a plot of the estimate of effect size in each study against an estimate of its precision (typically its standard error) [2]. We would expect the effect estimate in large studies with high precision to be close to the true effect, while studies with lower precision will have effect estimates evenly distributed on either side of the true effect, creating a funnelshaped plot. If there is publication bias, then studies with low precision that have negative or nonsignificant results will be missing from the plot because they were not published, producing a funnel plot that is asymmetric.
Identifying funnel plot asymmetry may therefore suggest the possibility of publication bias. It should be remembered, however, that funnel plot asymmetry may have causes other than publication bias: selective reporting bias, where studies are published but outcomes with results that were not statistically significant are not presented, can also result in an asymmetric funnel [3], as can cases where there are genuine differences in effect between smaller and larger studies. Of course, if there are differences in effect across studies, then a naïve metaanalysis may be inappropriate and further analyses, such as a metaregression, may be needed.
A problem with using a funnel plot to look for evidence of asymmetry and hence of publication bias is that it is purely visual. It is subjective and not a formal statistical test. This may lead to misinterpretation, particularly when there are few studies [5]. Readers may find it difficult to interpret funnel plots [6] and may be misled by the choice of axes or outcome measure [7]. Using funnel plot asymmetry to assess publication bias poses several problems including how we determine whether a plot is asymmetric, whether apparently asymmetric plots can occur by chance, how much asymmetry is required to have good evidence of bias, and how likely a plot is to be asymmetric when there is publication bias.
In this paper, we consider these issues by examining two new statistics for assessing funnel plot asymmetry: Imbalance and Asymmetry Distance. Metaanalyses with and without publication bias are simulated and the resulting funnel plots assessed for asymmetry using these new statistics. We show that apparently asymmetric funnel plots can frequently arise by chance and that even when there is considerable publication bias, funnel plots are rarely asymmetric.
Methods
Visual assessment of funnel plots
When examining a funnel plot for asymmetry, we might typically identify asymmetry by noticing an absence of studies on the lower corner of one side of the funnel plot and a corresponding greater number of studies in the opposite corner. This might suggest that smaller studies with unfavourable results were not published. For example, in Figure 1, there are fewer studies in the lower left corner of the funnel than the lower right corner. Hence, a simple way to assess asymmetry is to count the number of studies in each corner of the funnel plot and compare them. In Figure 1, when considering the bottom half of the funnel (where standard error is greater than 0.2), there are six studies on the lower right of the funnel but only two on the lower left side, a difference of four. For the purposes of this paper, we will call this difference in the number of studies on either side of the lower part of the funnel the ‘Imbalance’ in the funnel plot. How important any identified imbalance is must be interpreted relative to the total number of studies in the analysis.
That is, the difference in distances on the two sides of the funnel dived by the total distance. The Asymmetry Distance ranges from zero (perfect symmetry) to one (maximum asymmetry, all studies on same side of the funnel). This Asymmetry Distance is harder to judge by eye than the Imbalance, but it is easily calculated using the available effect estimates, or directly from a funnel plot. In Figure 1, the Asymmetry Distance among studies with a standard error greater than 0.2 is 0.71. We will consider how to interpret this result later in this paper.
While the Imbalance and Asymmetry Distance can represent how we might visually assess a funnel plot, and provide simple means to assess funnel plot asymmetry, it is not clear how large either of these should be in order to conclude that a funnel plot is asymmetric. We investigate this in a simulation study.
A simulation study
To simulate a metaanalysis, a set of S studies was sampled without replacement from the full set of 10,000 studies. The Imbalance and Asymmetry Distance, as defined above, were calculated for this metaanalysis of S studies considering only studies with σ _{i} ≥ 0.7 (the median standard error in the simulation), to represent assessing asymmetry based on studies at the bottom of the funnel plot. As Egger’s test for funnel plot asymmetry [8] is widely used to test for publication bias, Egger’s test was also performed to compare the visual assessments of asymmetry to a formal test. In this simulation study, metaanalyses with S = 5, 10, 15, 20 … up to 100 studies were considered. For each value of S a total of 10,000 simulated metaanalyses were tested. This simulation assessed the performance of the Imbalance and Asymmetry Distance in the absence of publication bias.
To simulate publication bias, the model of Copas and Shi was used [9]. This model has three parameters a, b and a correlation ρ. For each study in a simulated metaanalysis, a random variable δ _{i} is drawn from a standard normal distribution with the condition that \( \mathrm{corr}\left(\frac{\theta_{\mathrm{i}}}{\sigma_{\mathrm{i}}},{\delta}_{\mathrm{i}}\right)=\rho \) and the study is published only if \( a+\frac{b}{\sigma_{\mathrm{i}}}+{\delta}_{\mathrm{i}}>0 \). The choice of a, b and ρ determines the degree of publication bias. Four different cases of the model with varying values of a and b were considered, as shown in Figure 2, each with ρ = 1, to represent differing degrees of publication bias. The parameter ρ can be interpreted as the probability that a study in the lower corner of the funnel is not published. Setting ρ = 1 therefore ensures that there are regions of the funnel with no published studies at all, so this represents fairly extreme publication bias.
The simulation process described above was repeated, but this time sampling the S studies for the metaanalysis from the sets with publication bias shown in Figure 2, so the simulated metaanalyses were subject to publication bias. As before, 10,000 simulated metaanalyses were examined for each S in 5, 10, 15… 100. The simulation process was repeated for each of the four sets of studies in Figure 2, representing four differing degrees of publication bias.
Results
Results without publication bias
These results show that the Imbalance of 4 and the Asymmetry Distance of 0.71 for Figure 1 which has 19 studies are not sufficient to conclude that there is funnel plot asymmetry as this degree of asymmetry could have arisen by chance in the absence of publication bias.
Results with publication bias
This simulation confirms the low power of Egger’s test: achieving even 50% power requires either extreme publication bias or a large number of studies (for example, over 50).
The examples so far all represent fairly extreme cases of publication bias, where there were regions of the funnel plot with no publishes studies at all (Figure 2). In additional simulations studies, we found that more realistic instances of publication bias, where some studies in those regions were published (for example, by setting ρ = 0.8 in the Copas and Shi model, results are presented in Additional file 1), further reduce the power of both visual assessment of the funnel plot and Egger’s test to detect asymmetry.
This analysis also assumed no heterogeneity in the effect estimates across trials. We performed an additional simulation study by including a heterogeneity τ ^{2} of 0.25 (moderate heterogeneity with I ^{2} = 33%) when simulating study effect sizes. Including heterogeneity across the simulated studies does not affect the 95% centiles when there is no funnel plot asymmetry but does reduce the power of the funnel plot to detect asymmetry (results are presented in Additional file 2). We note that the power of Egger’s test when there is heterogeneity is lowered considerably more than the power for the Asymmetry Distance. This suggests that an inspection of the funnel plot may be preferable to Egger’s test when there is substantial heterogeneity.
Discussion
This study has shown that a visual assessment of a funnel plot in a metaanalysis is generally a poor method of assessing whether funnel plot asymmetry or publication bias is present. Unless the number of studies in the metaanalysis is very large, apparently asymmetric funnel plots frequently occur by chance when there is no underlying asymmetry. Even when there is considerable publication bias, funnel plots lack power to detect it because most funnel plots will not appear to be particularly asymmetric.
Both the Imbalance and the Asymmetry Distance are approximations designed to give numerical values that represent how one might view a funnel plot. More sophisticated consideration of a funnel plot might improve the ability to detect funnel plot asymmetry. The problem with visually assessing a funnel plot, however, is precisely that it is not a formal statistical test. As such, there is no clear way of determining how much asymmetry is required to reject the null hypothesis of a symmetric funnel plot.
The simulation has shown that high levels of Imbalance and Asymmetry Distance are required to reject the null hypothesis of no symmetry, and when there are ten or fewer studies in the metaanalysis (very common in medical contexts), even an Asymmetry Distance of 1 (the maximum possible) is insufficient to reject the null hypothesis. It is reasonable to conclude that any funnel plot, no matter how assessed, would have to appear very asymmetric before one could be confident that the asymmetry had not arisen by chance. So there is a substantial chance of type I error when visually assessing a funnel plot. Even the funnel plot illustrated in Figure 1, which has been used elsewhere as an example of funnel plot asymmetry [4], does not in fact have sufficient asymmetry, in terms of the Imbalance or Asymmetry Distance, to reject the possibility that the asymmetry arose by chance alone, despite being on the borderline of statistical significance using Egger’s test (P = 0.061).
Funnel plots have many problems in addition to those identified. How the funnel plot is presented, such as which metric is used for the vertical axis, can substantially change the appearance and hence the interpretation of the plot. Other research has shown that funnel plots are often interpreted inconsistently and that researchers are generally poor at interpreting funnel plots [6]. Funnel plot asymmetry, even if it is identified, is not necessarily evidence that there is publication bias. Heterogeneity in the effect estimates can lead to asymmetry, for example, in metaanalyses of clinical trials where small trials may be targeted to individuals more likely to benefit from treatment, whereas larger trials recruit patients from a more general population. In such cases, a naïve metaanalysis may be inappropriate and further analyses, such as a metaregression, may be needed.
Conclusions
Presenting a funnel plot in a metaanalysis may be highly misleading, particularly when then are ten or fewer studies in the analysis, as is common in metaanalyses of clinical trials. While this paper has presented two new statistics for assessing funnel plot asymmetry, it is not proposed that they be used because of this potential for misleading interpretation. Formal statistical test of asymmetry, such as Egger’s test, should generally be preferred to funnel plots because, although such tests lack statistical power, they have appropriate type I error. Funnel plots, if they are to be presented at all, may be best used as illustrations of asymmetry when there is good evidence from a formal test that funnel plot asymmetry is present.
Abbreviations
 AD:

Asymmetry Distance
Declarations
Acknowledgements
The author would like to thank Lesley Stewart for her assistance in proof reading and editing the manuscript.
No funding was sought or received for this article.
Authors’ Affiliations
References
 Dickersin K. The existence of publication bias and risk factors for its occurence. JAMA. 1990;263:1385–9.PubMedView ArticleGoogle Scholar
 Light RJ, Pillemer DB. Summing up: the science of reviewing research. Cambridge: Harvard University Press; 1984.Google Scholar
 Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, et al. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.PubMedView ArticleGoogle Scholar
 Raudenbush SW. Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: a synthesis of findings from 18 experiments. J Educ Psychol. 1984;76(1):85–97.View ArticleGoogle Scholar
 Lau J, Ioannidis JPA, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333(7568):597–600.PubMed CentralPubMedView ArticleGoogle Scholar
 Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005;58(9):894–901.PubMedView ArticleGoogle Scholar
 Tang JL, Liu JLY. Misleading funnel plot for detection of bias in metaanalysis. J Clin Epidemiol. 2000;53(5):477–84.PubMedView ArticleGoogle Scholar
 Egger M, Smith GD, Schneider M, Minder C. Bias in metaanalysis detected by a simple, graphical test. BMJ. 1997;315(7109):629–34.PubMed CentralPubMedView ArticleGoogle Scholar
 Copas J, Shi JQ. Metaanalysis, funnel plots and sensitivity analysis. Biostatistics. 2000;1(3):247–62.PubMedView ArticleGoogle Scholar
Copyright
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.