Sample size and power considerations in network meta-analysis
© Thorlund and Mills; licensee BioMed Central Ltd. 2012
Received: 17 May 2012
Accepted: 12 September 2012
Published: 19 September 2012
Skip to main content
© Thorlund and Mills; licensee BioMed Central Ltd. 2012
Received: 17 May 2012
Accepted: 12 September 2012
Published: 19 September 2012
Network meta-analysis is becoming increasingly popular for establishing comparative effectiveness among multiple interventions for the same disease. Network meta-analysis inherits all methodological challenges of standard pairwise meta-analysis, but with increased complexity due to the multitude of intervention comparisons. One issue that is now widely recognized in pairwise meta-analysis is the issue of sample size and statistical power. This issue, however, has so far only received little attention in network meta-analysis. To date, no approaches have been proposed for evaluating the adequacy of the sample size, and thus power, in a treatment network.
In this article, we develop easy-to-use flexible methods for estimating the ‘effective sample size’ in indirect comparison meta-analysis and network meta-analysis. The effective sample size for a particular treatment comparison can be interpreted as the number of patients in a pairwise meta-analysis that would provide the same degree and strength of evidence as that which is provided in the indirect comparison or network meta-analysis. We further develop methods for retrospectively estimating the statistical power for each comparison in a network meta-analysis. We illustrate the performance of the proposed methods for estimating effective sample size and statistical power using data from a network meta-analysis on interventions for smoking cessation including over 100 trials.
The proposed methods are easy to use and will be of high value to regulatory agencies and decision makers who must assess the strength of the evidence supporting comparative effectiveness estimates.
Over the past 2 decades, meta-analysis has become increasingly accepted by clinicians, decision-makers and the public as providing high-quality assessments of evidence . Network meta-analysis, a new expansion of meta-analysis that allows for simultaneous comparison of several treatments, is similarly becoming increasingly accepted in the clinical research community [2–11]. Having been available for more than 3 decades, meta-analysis has been studied extensively, and several hundred articles published in this period have identified and resolved a vast array of basic and advanced methodological issues [1, 12]. Network meta-analysis inherits all the challenges present in a standard meta-analysis (e.g., issues of bias, heterogeneity and precision), but with increased complexity due to the multitude of comparisons involved [5, 13]. Since network meta-analysis is still a relatively new technique, the number of publications addressing methodological challenges is still relatively sparse.
One important issue that has received much attention in individual trials and meta-analysis is the issue of sample size and statistical power [14–26]. Several studies have demonstrated the importance of interpreting pooled meta-analysis estimates and confidence intervals according to the statistical level of evidence (i.e., precision) [14, 15, 18–24, 26, 27], and sound recommendations have been provided [16, 22, 28]. So far, however, only a small number of studies have addressed the issue of power and precision in network meta-analysis [8, 13, 29], and no comprehensive guidance exists on the topic. Network meta-analyses typically include many more trials than standard meta-analyses because of the multitude of comparisons involved and for this reason may artificially appear to provide a stronger evidence base. Likewise, the accompanying graphical representation of a treatment network can provide a similar compelling but potentially false impression of a strong evidence base.
Network meta-analysis utilizes evidence from direct (head-to-head) comparisons (i.e., trials directly comparing treatment A and B) and indirect comparisons (e.g., the combination of trials comparing A with C and trials comparing B with C) [4, 6, 30]. The major challenge in interpreting the power and precision of a network meta-analysis stems from the fact that there are (typically) varying levels of power and precision across all comparisons. In addition, the power and precision of indirect evidence are more complex to assess than for direct evidence, and thus, without proper guidance, it will be difficult for most authors to evaluate the precision gain from use of indirect evidence as well as the strength of evidence in a treatment network.
In this article, we provide guidance on quantifying the power and precision in network meta-analysis using simple sample size considerations. We first describe how to quantify the precision in indirect comparison meta-analysis and subsequently in network meta-analysis with combinations of direct and indirect evidence. We then outline the concept of sample size requirements and power calculations in pairwise meta-analysis. Finally, we show how to combine these measures in order to quantify the power and strength of evidence available for all treatment comparisons in a network meta-analysis. We illustrate the described methods using data from a recent network meta-analysis on interventions for smoking cessation .
(Note, when dealing with ratio effect measures, such as relative risks and odds ratios, all calculations are done on the log scale to preserve linearity and approximate normality). To produce confidence intervals for the indirect estimate, we first need to estimate its variance. The variance of in the indirect estimate of A versus B (V AB ) is simply equal to the sum of the variance of the effect estimate of A versus C (V AC ) and the variance of the effect estimate of A versus B (V AB ). Mathematically this corresponds to the equation V AB = V AC + V BC . It is therefore clear that the variance of a (direct) meta-analysis effect estimate based on some number of trials, say k, will always be smaller than the variance of an indirect meta-analysis based on the same number of trials, k (all trial sample sizes being equal). In other words, direct estimates come with higher precision and power (trial count and trial sample sizes being equal). In many situations, however, using indirect estimates can add considerable power and precision.
When both direct and indirect evidence is available, it may often be advantageous to combine the two statistically [2, 4–7, 30, 33]. For example, if only two small trials have investigated two active interventions A and B head to head, but 20 trials have compared A or B with placebo, the indirect evidence will be able to add much power and precision to the comparative estimate of A and B. The combination of indirect and direct evidence requires advanced statistical regression techniques (i.e., network meta-analysis) that are beyond the scope of this article [4, 6, 30]. However, in the context of sample size and power considerations, it suffices to understand that indirect evidence, when combined with direct evidence, increases the power and precision of treatment effect estimates [4, 6, 7, 9, 30, 33]. The extent to which is does so can be evaluated readily by using the methods we describe below.
In this section we introduce three methods for gauging how much statistical precision an indirect estimate provides when no direct evidence is available. In particular, we describe how to approximate the amount of information required in a direct (head-to-head) meta-analysis to produce the same precision as that in the available indirect evidence. Simply put, what direct meta-analysis sample size would provide a similar degree of information? We dub this the effective sample size of the indirect evidence or, interchangeably, the effective indirect sample size. We describe three different methods for approximating the effective indirect sample size. Each of these methods differs with respect to simplicity and validity (the simpler one being the least valid), so we outline the simplicity-validity trade-offs at the end of the section.
The required number of indirect comparison trials required to produce the same precision as a given number of direct (head-to-head) trials
Trial count ratio
Exact precision ratio
Number of indirect comparison trials required to match precision from the corresponding number of single trials in a pair wise meta-analysis
16 (4:12 )
The cells in underlined italics indicate where the indirect evidence produces the exact precision of the corresponding number of trials. The remaining cells indicate where the indirect evidence produces precision slightly above that of the corresponding number of head-to-head trials.
In some cases the required number of trials in an indirect comparison for a specific trial count ratio produces a precision corresponding to more than that of a stated number of single head-to-head trials. For example, with a trial count ratio of 1:3, one would require 2 × 5.33 = 10.66 indirect comparison trials to produce the precision of two head-to-head trials. However, since we cannot have fractions of trials, we take the closest integer above 10.66 where the trial count ratio is maintained: 12 trials with a trial count ratio of 3:9.
Table 1 can readily be used for quickly and easily checking how many head-to-head trials the indirect evidence ‘effectively’ corresponds to. That is, if the indirect evidence produces the precision of, say, three trials, we can think of the evidence as being as strong as a meta-analysis of three head-to-head trials. For example, if one has an indirect comparison with 4 trials comparing A with C, and 12 trials comparing B with C, the precision of the indirect comparison corresponds to a meta-analysis of 3 trials directly comparing A with B (Table 1). It should be noted that Table 1 is only valid to the extent that trial sample sizes and trial population variances are similar across trials, as well as the extent to which heterogeneity is absent or ignorable.
Another relatively simple approach to gauging the degree of power and precision from an indirect comparison is to consider the collection of trials included in each comparison as one (large) clinical trial. From a sample size perspective, following similar mathematic derivations as the above trial count perspective, the relationship between the precision of an indirect comparison and the precision of a direct meta-analysis turns out the same (see Appendix 1.b for mathematical derivations). For example, to produce the same precision as a head-to-head meta-analysis including 1,000 patients, one would need a total of 4,000 (4 × 1,000) patients in the indirect comparison, provided the number of patients is the same for the two comparisons (2,000:2,000). Taking the sample size perspective comes with the flexibility of a possible reversal of the calculation. For example, an indirect comparison with 500 patients in the A vs. C comparison and 500 patients in the B vs. C comparisons would produce the same precision as a direct comparison with 250 patients [(500 + 500)/4]. Likewise, in a scenario with 1,000 patients in comparison A vs. C, and 10,000 patients in comparison B vs. C, the exact precision ratio is 12.1 (see Table 1), and so the effective direct meta-analysis sample size would be (1,000 + 10,000)/12.1 = 909.
Effective heterogeneity-corrected sample sizes of indirect comparison scenarios with varying degrees of patient count ratios and heterogeneity in each comparison (A vs. C and B vs. C), but with fixed total sample size of 10,000
I2adjusted precision ratio
Effective sample size
For the above example, the effective sample size using this formula is therefore
The above simple approach to calculate the effective indirect sample size does not consider the possibility that statistical heterogeneity exists across trials. When heterogeneity is present, the effect estimates that go into the indirect estimate incur a higher degree of variation, and so the effective indirect sample size corresponding to a head-to-head meta-analysis will be smaller than with the above simple approach. In line with already-established heterogeneity corrections for meta-analysis required sample sizes [23, 24, 28], we put forward that the actual number of patients in each of the comparisons informing the indirect estimate can be penalized by the additional variation explained by heterogeneity. In line with previous proposals, we penalize for the ‘lack of homogeneity’ [23, 24] using the popular measure of heterogeneity, I 2 , as a basis for the penalization.
Consider the example where a meta-analysis of A vs. C includes 6,000 patients and a meta-analysis of B vs. C includes 8,000 patients, and assume the estimated degree of heterogeneity for A vs. C is 50% (I AC 2 = 50%) and 25% for B vs. C (I BC 2 = 25%). Then the lack of homogeneity is 100%-50% = 50% for A vs. C and 100%-25% = 75% for B vs. C. We penalize the actual sample size by multiplying the actual sample size by the lack of homogeneity, so that the penalized sample size of A vs. C is 50% × 6,000 = 3,000, and the penalized sample size for B vs. C is 75% × 8,000 = 6,000. The total penalized number of patients in the indirect comparison is then 3,000 + 6,000 = 9,000, the patient count ratio is 1:2 (same as 3,000:6,000), the precision ratio is 4.5 (see Table 1), and so the effective heterogeneity-corrected sample size in this indirect comparison is 9,000/4.5 = 2,000.
where n AC and n BC are the actual sample sizes (before correction) in the meta-analyses of A vs. C and B vs. C, respectively, and where the precision ratio is based on the heterogeneity-corrected sample sizes (see Table 1 and Table 2). In the appendix we provide a general formula for the precision ratio.
One immediate limitation of the above-proposed sample size heterogeneity correction is the fact that I 2 s are typically unreliable and unstable in meta-analyses including a limited number of trials and will depend on the effect metric used [34–36]. In most cases, it will therefore be preferable to simply assume some plausible degree (percentage) of heterogeneity derived from a mix of clinical considerations and the I2 estimate at hand. Typically, an assumption of 25% or 50% heterogeneity will be reasonable in the context of sample size considerations . Table 2 illustrates the effective sample size for various indirect comparison scenarios including a total of 10,000 patients, under different combinations of heterogeneity corrections (one including no correction).
Another limitation of the sample size approach is the inherent assumption that sample size is a good proxy for precision. This may not be true if there are some important differences in event rates (for binary data) or counts (count data), or if there are population differences in trials that result in notably different standard deviations, but not necessarily different effect estimates. To curb this limitation for binary data, one may for example choose to focus on the effective number of events. A more universally applicable approach focuses on a measure called statistical information. We describe this below.
Statistical information, also sometimes referred to as Fisher information, is a more complex statistical measure for gauging the degree of precision present in a data set. For pairwise meta-analyses, the statistical information is equal to the inverse of the pooled variance (i.e., one divided by the variance), which is also the measure of precision . For indirect comparisons, the statistical information (precision) is equal the inverse of the pooled indirect variance. That is, with variances V AC and V BC for comparisons A vs. C and B vs. C, the indirect variance is V AC + V BC , so indirect statistical information is 1/(V AC + V BC ). Because the variance incorporates heterogeneity and is dependent on the number of trials and the sample size, no further calculations or adjustments are needed.
A major disadvantage of statistical information in meta-analysis is that it operates on a scale that no statistically non-sophisticated reader could easily grasp . The statistical information may however be useful in the context of sample size requirements since it is possible to calculate the required statistical information (analogous to the required sample size) and compare the actual statistical information present in an indirect comparison with such a yardstick for sufficient power.
Strengths and limitations of the three approaches for gauging the effective degree of power and precision in indirect comparisons
Effective number of trials
1. Easy and fast to calculate
1. Only valid to the extent trial sample sizes are equal and heterogeneity is absent
2. Lacks flexibility for approximate trial count ratios
Effective sample size (ignoring heterogeneity)
1. Easy and fast to calculate
1. Does not account for heterogeneity
2. Exact calculations for all trial count ratios
2. Assumes equal meta-analysis population variances across comparisons
3. Sample size (no. of patients) resonates well with clinicians
Effective sample size (correcting for heterogeneity)
1. Exact calculations for all precision ratios
1. Assumes equal meta-analysis population variances across comparisons
2. Accounts for heterogeneity
2. Depends on precise heterogeneity estimation
3. Easy to calculate
4. Sample size (no. of patients) resonates well with clinicians
1. Statistical information does not resonate well with a clinical audience
Effective statistical information
1. Theoretically statistically exact
2. Not straight forward to calculate
3. Depends on precise heterogeneity variance estimation or requires elicitation of Bayesian variance priors
So far we have only described situations where the indirect estimate is obtained through one common comparator. However, it is worth noting that indirect evidence may often come from scenarios where the link between two treatments of interest must go through two or more comparators. For example, if we wish to compare A with D and the following three comparisons are available—A vs. B, B vs. C and C vs. D—then the link between A and D goes through both B and C. The indirect variance is now a sum of three (rather than two) direct variances, . We would therefore expect comparably smaller precision. Calculating the effective number of trials in the two common comparator example, we find that with an equal number of trials, say 3:3:3, the precision of the indirect comparison corresponds to that of only one direct trial, that is, an exact precision ratio of 1:9. With a more unbalanced number of trials, say 8:1:8, we get an exact precision ratio of 1:21.
Such consistently large direct to indirect precision ratios indicate that indirect comparisons with two (or more) comparators in the link will typically add very little precision. For this reason we dub them ‘weak links.’ In the context of combining direct and indirect evidence as we address below, it seems that weak links will typically only add an ignorable small amount of precision to the final estimate and thus, for simplicity, may be ignored for sample size and power considerations in indirect comparisons and network meta-analysis.
The simplest example of a combination of direct and indirect evidence is the three-treatment loop where precision is added to the comparison of A and B by borrowing strength from an indirect comparison based on some common comparator C. Whether our measure of precision (information) is the number of trials, the sample size or the statistical information, the total amount of precision available for a particular comparison in a three-treatment loop is conceptually the sum of information in the direct evidence and in the indirect evidence.
To calculate the effective number of trials informing a particular comparison (e.g., A vs. B) in the combined evidence, we simply add the effective number of trials in the indirect evidence to the number of trials in the head-to-head evidence.
To calculate the effective number of patients informing a particular comparison, we simply add the effective number of patients in the indirect evidence to the number of patients in the head-to-head evidence. If heterogeneity adjustments have been applied for the indirect evidence, a similar adjustment should be applied to the direct evidence (we illustrate this in our worked example below).
To calculate the statistical information informing a particular comparison, we simply take the sum of the inverse indirect variance and the inverse direct variance. Alternatively, the statistical information may be extracted directly from the statistical software package used to run the network meta-analysis. (Note that if multi-arm trials are included in the analysis, some adjustments for correlations are needed).
In many treatment networks two or more sources of indirect evidence informing some (or all) of the comparisons may exist. For example, two active treatments A and B may both have been compared to standard-of-care and placebo. In this case, indirect evidence exists from two sources. Extending the three-treatment example above, estimating the total effective amount of information is simply a task of summing all indirect and direct evidence.
In a similar situation, multiple sources of indirect evidence may exist where no direct evidence exists. In this case, the effective number of trials, sample size or statistical information is simply obtained by summing all indirect information.
We previously discussed ‘weak links.’ Since these add a relatively small amount of information, one may chose to ignore them without notable loss of information (but with ease in the calculations). This goes for both situations where direct evidence and indirect evidence are being combined, and where there are multiple sources of indirect evidence.
Before getting into sample size and power considerations for indirect comparison meta-analysis and network meta-analysis, we first outline the already well-established framework for pairwise meta-analysis. We then extend the concept to indirect comparison meta-analysis and lastly to network meta-analysis.
Here z 1-α/2 and z 1-β are the (1-α/2)th and (1-β)th percentiles of a standard normal distribution, and C is a constant depending on the randomization ratio and number of treatment arms (C = 4 with a randomization ratio of 1:1 and two treatment arms).
If statistical heterogeneity exists across the included trials in a meta-analysis, one can adjust the calculated sample size to account for the additional variation (i.e., increased uncertainty) [21–24, 28]. This is achieved by multiplying the required sample size, N, by a heterogeneity correction factor 1/(1-H), where H has a similar interpretation as the well-known measure I 2 (the percentage of variation in a meta-analysis explained by heterogeneity) and is the a priori or maximum acceptable degree of heterogeneity [21–24, 28]. Empirical evidence and simulations have demonstrated that such adjustments perform well in maintaining the desired statistical power [21, 22, 24].
A simulation study has demonstrated adequate performance of the required statistical information when the heterogeneity is modeled with certain Bayesian priors .
This measure gives us an idea of how far we have come and how much farther of a distance there is to the yardstick—our required sample size.
where Φ is the cumulative standard normal distribution function.
So far we have described method for estimating the effective sample size in indirect and combined evidence, as well as methods for estimating the required sample size and gauging the strength of evidence in pairwise meta-analyses. We now combine these measures to evaluate the strength of indirect comparisons and network meta-analyses. For the remainder of this article we concentrate on the number of patients, but our example and calculations can easily be carried out using the number of trials or statistical information.
We have introduced the effective sample size for indirect evidence. In line with its definition, we can derive the effective indirect information fraction and the effective power in an indirect comparison. The steps required to do this are as follows. First, calculate the effective indirect sample size. Second, calculate the required sample size for a direct meta-analysis. Third, to get the effective indirect information fraction, simply divide the effective number of patients in the indirect comparison by the required sample size for a direct comparison. Fourth, to calculate the power of the available indirect evidence, simply insert the effective indirect sample size as the n in the above formula.
The steps required to calculate the information fraction and power of treatment comparisons that are both informed by direct and indirect evidence or by multiple sources of indirect evidence are similar to the step required for indirect comparisons. First, calculate the effective sample size for the comparison of interest by summing up the evidence from the available sources (we described how to do this above). Second, as before, calculate the required sample size for a direct meta-analysis. Third, as before, calculate the effective information fraction by dividing the effective number of patients by the required sample size. Fourth, calculate the power of the available evidence by inserting the effective sample size as the n in the above formula.
A recent MTC explored the efficacy of five different interventions for smoking cessation, low-dose NRT (<22 mg nicotine patches), which is an over-the-counter intervention, and four newer and more expensive interventions, combination NRT (i.e., patch plus gum), high-dose NRT (>22 mg patches), buproprion and varenicline .The published MTC provides a useful data-set to illustrate the issues we have raised in this article. Low-dose NRT is already well established and well known to yield about a 50% higher success rate than inert control interventions (i.e., relative risk of approximately 1.50). To consider any of the four newer treatments worthwhile, one could argue for the need to demonstrate at least an additional 20% efficacy. These considerations make up the basis for the required sample size considerations. Keep in mind that the published MTC has much higher rates of effective sample size than the assumptions we make in this example. 
The effective sample sizes and corresponding information fractions and power estimates from the four comparisons of newer treatments vs. low-dose NRT
Comparison (vs. low-dose NRT)
Effective head-to-head sample size
Effective indirect sample size
Total effective sample size
Statistical power to detect observed improvement*
Network meta-analysis odds ratio estimate (95% credible interval)
1.05 (0.76 – 1.41)
1.32 (1.11 – 1.57)
0.99 (0.86 – 1.14)
1.38 (1.15 – 1.64)
In the network meta-analysis of interventions for smoking cessation, high-dose NRT and varenicline were both significantly better than low-dose NRT and demonstrated effect estimates larger than the a priori considered minimally important difference. For these comparisons, high-dose NRT is supported by 88% power, and varenicline is supported by 76% power. Considering that the true effects of high dose NRT and varenicline over low dose NRT are much higher than the 20% increase that was assumed for these calculations, the true power of these comparisons are also much higher. For example, if the MTC actual effects for varenicline (i.e., a 38% increase in smoking cessation compared with low dose NRT) are used, the statistical power to detect this difference then exceeds 99%. Combination NRT and buproprion both yielded effect estimates very similar to low-dose NRT. Considering that the two are supported by 66% and 95% power, respectively, and that none of the two effect estimates appear superior, it should be reasonable to infer that the two interventions do not offer any noteworthy benefit over low-dose NRT.
In this article we have outlined available methods for gauging the strength of the evidence in a network meta-analysis using sample size and power considerations. We recommend sample size considerations in the context of the number of patients, as the required calculations are relatively straightforward and will resonate well with most clinicians and decision-makers. The methods we have outlined are of high value to regulatory agencies and decision makers who must assess the strength of the evidence supporting comparative effectiveness estimates.
1.a Calculating the effective number of trials
This formula creates the basis for the results presented in Table 1.
1.b Calculating the effective number of patients
Consider the situation where three treatments, A, B and C, have been compared head to head in randomized clinical trials. Assume that the population variance of comparative treatment effects is the same for A vs. B, A vs. C and B vs. C, and assume the population variance produced by a fixed-effect pairwise meta-analysis can be regarded as a large well-designed clinical trial. Let n AB , n AC and n BC denote the meta-analysis sample size (total number of patients) for the three comparisons A vs. B, A vs. C and B vs. C, respectively.
and subsequently use these penalized sample sizes in the formula for the effective indirect sample size.
2. Information fraction and power calculations – worked example
This total effective sample sizes correspond to information fractions of 92% and 60% and statistical power estimates of 88% and 72%.
This total effective sample sizes correspond to information fractions of >100% and 60% and statistical power estimates of 95% and 71%.
All power, and information fraction calculations above are geared to detect an assumed relative improvement in smoking cessation of 20%. All calculations are highly sensitive to the assumed relative improvement. In particular, assuming larger improvements would result in substantially larger power and information fraction estimates.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.