Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study
 Elie A Akl^{1, 2}Email author,
 Lara A Kahale^{1},
 Arnav Agarwal^{2},
 Nada AlMatari^{1},
 Shanil Ebrahim^{2, 5, 8},
 Paul Elias Alexander^{2},
 Matthias Briel^{2, 3},
 Romina BrignardelloPetersen^{2, 4},
 Jason W Busse^{2, 5, 6},
 Batoul Diab^{1},
 Alfonso Iorio^{2, 16},
 Joey Kwong^{9},
 Ling Li^{9},
 Luciane Cruz Lopes^{7},
 Reem Mustafa^{2, 10},
 Ignacio Neumann^{2, 11},
 Kari AO Tikkinen^{12},
 Per Olav Vandvik^{13, 14},
 Yuqing Zhang^{2},
 Pablo AlonsoCoello^{15} and
 Gordon Guyatt^{2, 16}
DOI: 10.1186/204640533137
© Akl et al.; licensee BioMed Central Ltd. 2014
Received: 30 October 2014
Accepted: 13 November 2014
Published: 26 November 2014
Abstract
Background
There is no consensus on how authors conducting metaanalysis should deal with trial participants with missing outcome data. The objectives of this study are to assess in Cochrane and nonCochrane systematic reviews: (1) which categories of trial participants the systematic review authors consider as having missing participant data (MPD), (2) how trialists reported on participants with missing outcome data in trials, (3) whether systematic reviewer authors actually dealt with MPD in their metaanalyses of dichotomous outcomes consistently with their reported methods, and (4) the impact of different methods of dealing with MPD on pooled effect estimates in metaanalyses of dichotomous outcomes.
Methods/Design
We will conduct a methodological study of Cochrane and nonCochrane systematic reviews. Eligible systematic reviews will include a grouplevel metaanalysis of a patientimportant dichotomous efficacy outcome, with a statistically significant effect estimate. Teams of two reviewers will determine eligibility and subsequently extract information from each eligible systematic review in duplicate and independently, using standardized, prepiloted forms. The teams will then use a similar process to extract information from the trials included in the metaanalyses of interest. We will assess first which categories of trial participants the systematic reviewers consider as having MPD. Second, we will assess how trialists reported on participants with missing outcome data in trials. Third, we will compare what systematic reviewers report having done, and what they actually did, in dealing with MPD in their metaanalysis. Fourth, we will conduct imputation studies to assess the effects of different methods of dealing with MPD on the pooled effect estimates of metaanalyses. We will specifically calculate for each method (1) the percentage of systematic reviews that lose statistical significance and (2) the mean change of effect estimates across systematic reviews.
Discussion
The impact of different methods of dealing with MPD on pooled effect estimates will help judge the associated risk of bias in systematic reviews. Our findings will inform recommendations regarding what assumptions for MPD should be used to test the robustness of metaanalytical results.
Keywords
Missing participant data Imputation Risk of bias Trials Systematic reviews MetaanalysisBackground
Missing data for the outcomes of interest is a common problem in randomized trials[1]. In one study, almost one in every three trials with statistically significant results lost statistical significance when making plausible assumptions about the outcomes of participants with missing data[1]. This reduces our confidence in the effect estimates resulting not only from these trials but also from systematic reviews including them.
One challenge with abstracting data from trial reports is understanding whether or not data from a specific category of participants is missing. For example, a trial report might indicate that a certain number of participants withdrew consent, without indicating whether or not they continued to be followedup. In these cases, systematic review authors might need to make assumptions based on their best guess.
Moreover, a recent methodological survey found that systematic reviews are deficient in terms of dealing with and assessing the risk of bias associated with missing participant data (MPD) (unpublished data by Akl et al.)[2]. The Cochrane Collaboration’s Risk of Bias (RoB) tool was designed to help in assessing bias associated with a number of factors, including MPD[3]. In a recent study evaluating stakeholders’ experiences with, and perceptions of, the Cochrane RoB tool participants cited incomplete outcome data as one of the most difficult domains to assess[4]. They also requested more guidance on how to incorporate RoB assessments into metaanalyses and conclusions[4].
One approach to assessing the extent of risk of bias associated with MPD is to conduct sensitivity analyses based on different assumptions regarding the outcomes of participants with missing outcome data[5–7]. Examples of these assumptions include none, and all or a specified proportion of participants in each group suffering the outcome of interest. No study has thus far assessed how systematic reviewers actually deal with MPD. Similarly, there is a lack of evidence about the impact of different approaches for dealing with MPD on pooled effect estimates. This leaves uncertain the extent to which results of systematic reviews are vulnerable to MPD.
Objectives
The objectives of this study are to assess in Cochrane and nonCochrane systematic reviews: (1) which categories of trial participants the systematic review authors consider as having MPD, (2) how trialists reported on participants with missing outcome data in trials, (3) whether systematic reviewer authors actually dealt with MPD in their metaanalyses of dichotomous outcomes consistently with their reported methods, and (4) the impact of different methods of dealing with MPD on pooled effect estimates in metaanalyses of dichotomous outcomes.
Methods/Design
We did not register this protocol with PROSPERO given the register does not accept methodological reviews.
Definitions
MPD refers to outcome data for trial participants that are not available to the systematic reviewer authors (from the published trial reports or personal contact with trial authors) for inclusion in their metaanalyses.
MPD do not relate to any of the following:

Missing studies (e.g., unpublished studies);

Entire unreported outcomes (e.g., outcomes planned in trial protocols but not included in trial reports).
Cochrane systematic reviews are defined as systematic reviews published in the Cochrane Database of Systematic Reviews. All other systematic reviews will be considered nonCochrane systematic reviews.
A patientimportant outcome is defined as an outcome for which one would answer with “yes” to the following question: “if the patient knew that this outcome was the only thing to change with treatment, would the patient consider receiving this treatment if associated with burden, side effects, or cost?” We will use a previously developed hierarchy of outcomes for the selection of one outcome of interest (Additional file1)[8]. Categories I, II, and III include patientimportant outcomes. Category IV includes surrogate outcomes, which are not considered as patientimportant. For a composite outcome to be considered as patientimportant, we will require all of its components to be patientimportant[8].
Categories of participants with potential MPD
We will use the following mutually exclusive categories of participants that could be potentially counted as having MPD, at the trial level: “mistakenly randomized and inappropriately excluded,” “did not receive any treatment,” “withdrew consent,” “outcome not assessable,” “dead,” “experienced adverse events,” “noncompliant,” “discontinued prematurely,” “crossover,” “moved out of country,” and “lost to followup” (lost to followup for reasons not considered in our other categories). When it is not clear whether the participants in those categories have MPD or not, we will assume the following:

“Mistakenly randomized and inappropriately excluded,” and “did not receive any treatment” participant categories, which are defined prior to the initiation of the trial intervention, most likely have MPD;

“Withdrew consent,” “outcome not assessable,” “moved out of country,” and “lost to followup” (for reasons not considered in our other categories) participant categories, which are defined after the initiation of the trial intervention, most likely have MPD;

“Dead,” “experienced adverse events,” “noncompliant,” “discontinued prematurely,” and “crossover” most likely do not have MPD.
The number of participants with missing data will be the sum of the number of participants in each of the categories of participants known or inferred to have missing data.
Eligibility criteria
 1.Meets the following minimum criteria we set for a systematic review of trials:
 a.
Described as “systematic review” and/or “metaanalysis” of trials;
 b.
Compares a clinical intervention to another (or to no intervention);
 c.
Reports a search strategy of at least one database;
 a.
 2.
Addresses a preventive or a therapeutic clinical question in humans (diagnostic, prognostic, public health, and health service questions are not eligible);
 3.
Is either a Cochrane review or a nonCochrane review published in one of the core clinical journals;
 4.Includes a metaanalysis that meets the following criteria:
 a)
Is a grouplevel metaanalysis of randomized controlled trials and/or controlled clinical trials (e.g., network metaanalysis, Bayesian metaanalysis, and metaregression are not eligible);
 b)
Reports an effect estimate expressed as a dichotomous measure (including relative risk (RR) or odds ratio (OR); excluding those produced by generic inverse variance method);
 c)
Reports a statistically significant pooled effect estimate from at least two trials for a patientimportant efficacy outcome; statistical significance refers to p value < 0.05 or confidence interval (CI) not including 1.0.
 a)
 1.
A systematic review that is a duplicate publication (e.g., a Cochrane systematic review published in both the Cochrane Library and in a peerreviewed journal)
 2.
A metaanalysis with more than 20 included trials, for feasibility purposes.
 3.
A metaanalysis not reporting the numerical data used or each included trial (i.e., numerator and denominator per arm).
Search strategy
We will search for Cochrane systematic reviews in the Cochrane Library. We will use the Ovid MEDLINE interface to search for nonCochrane systematic reviews in the Core Clinical Journals (119 English language clinical journals indexed under Abridged Index Medicus by the National Library of Medicine (available at http://www.nlm.nih.gov/bsd/aim.html). The Abridged Index Medicus was initiated in 1970 to enable direct access to selected biomedical journals of interest to practicing physicians. We will restrict the search to 1 year but will not impose any language restrictions. Additional file2 provides the details of the search strategy.
Random sampling of citations
We will retrieve two random samples of Cochrane and nonCochrane systematic reviews from the pool of citations identified by our search. We will screen these two samples for eligibility using the above criteria. We will repeat the random sampling process as needed until reaching the final sample size, which will include the same number of Cochrane and nonCochrane systematic reviews (see Sample size section).
Selection process
We will conduct title and abstract screening, fulltext screening, and data abstraction in teams of two reviewers working independently and in duplicate. At the title and abstract screening stage, we will obtain the full text for any citation included by at least one reviewer. At the fulltext screening and data abstraction stages, the reviewers will resolve discrepancies by consensus, and if unsuccessful, with the help of a third reviewer. We will carry out calibration exercises at each level of the process for the purpose of verifying the validity and consistency of the review process. We will also develop and pilot test standardized data abstraction forms with instructions. A core group will communicate regularly to discuss progress and potential difficulties. A study flow will be developed to describe the results of the different steps of the selection process.
If a systematic review reports on more than one pairwise comparison with eligible metaanalyses, we will select the first one reported in the main text. If the selected comparison includes more than one eligible metaanalysis, we will select the one with the outcome that ranks the highest on the outcome hierarchy (Additional file1). If more than one outcome ranks the same on the outcome hierarchy, we will select the one reported first in the main text.
Data abstraction
We will conduct data abstraction using webbased systematic review software (DistillerSR™). In a first phase, we will collect data from the eligible systematic reviews and the eligible metaanalyses. In the second phase, we will collect data from trials included in the eligible metaanalyses.
Phase 1: we will abstract the following data from each eligible systematic review:
Characteristics of the systematic review:

Type of systematic review (i.e., Cochrane vs. nonCochrane);

Use or nonuse of the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach for assessing the confidence in effect estimates by outcome;[9]

Source of funding (e.g., private for profit, private not for profit, governmental, not funded, not reported).
Characteristics of the comparison of interest:

Type of intervention and control (e.g., pharmacological, surgical);

Outcome ranking on patientimportance hierarchy (i.e., I, II, III);

Duration of followup for outcome measurement;

Type of outcome (e.g., efficacy, safety).
Missing participant data:
In reference to the comparison and outcome addressed in the eligible metaanalysis, we will verify whether the systematic review:

Documented the categories of participants that could potentially be counted as having MPD (see previous section “Categories of participants with potential MPD”), and when documented, the number within each category;

Reported number of MPD and at what level (e.g., at the study arm level, study level, across studies);

Explicitly stated using the following for their metaanalysis: intentiontotreat, modified intentiontotreat, per protocol, or as treated;

Explicitly stated analyzing participants in the group to which they were randomized;

Explicitly stated the analytical method for dealing with MPD in the eligible metaanalysis (i.e., to account for MPD when generating the best effect estimate): complete case analysis, making assumptions for MPD, using the assumption used by the trialists, or excluding trials with high rate of MPD. Tables 1 and2 present the numerical details of the different assumptions that could be used to deal with MPD;
Numerical information from each trial to be used in the imputation analyses
Intervention arm  Control arm  

Number of participants randomized  Number of participant with missing data  Number of observed events  Number of participants randomized  Number of participant with missing data  Number of observed events  
Number of trial participants  a  b  c  e  f  g 
Numerical details of different methods to be used in the imputation analyses
Analytic method  Intervention arm  Control arm  

Numerator  Denominator  Numerator  Denominator  
Complete (available) case analysis  c  a  b  g  e  f 
None has event  c  a  g  e 
All had event  b + c  a  f + g  e 
Best case scenario  c  a  f + g  e 
Worst case scenario  b + c  a  g  e 
Using the concept of RI_{LTFU/FU}  [b x y x c/(a  b)] + c  a  [f x z x g/(e  f)] + g  e 
Incidence for missing participants same as observed in same arm^{a}  [b x c/(a  b)] + c  a  [f x g/(e  f)] + g  e 
Incidence for missing participants in both arms same as observed in control arm  [b x g/(e  f)] + c  a  [f x g/(e  f) ] + g  e 

Explicitly stated any additional metaanalyses for dealing with MPD for the outcome of interest (“reported analytical method”) (i.e., to judge the risk of bias associated with MPD): complete case analysis, making assumptions for MPD, using the assumption used by the trialists, or excluding trials with high rate of MPD (see Tables 1 and2);

Explicitly stated using assumptions for MPD accounted for uncertainty associated with imputing events;

Justification for the analytical method used to deal with MPD in the eligible metaanalysis;

Tool(s) used to judge risk of bias associated with MPD at the study level (e.g., Cochrane RoB tool), if any;

Method(s) used to assess risk of bias associated with MPD at the metaanalysis level (e.g., sensitivity analysis, subgroup analysis), if any.
Data from the eligible metaanalysis:

Number of trials included;

Numerator and denominator used in the metaanalysis for each arm for each trial;

Number of MPD reported (per arm or combined);

Pooled relative effect measure (RR or OR) and its associated CI, p value, and measure of heterogeneity (I ^{2});

Analysis model used (i.e., random effect or fixed effect);

Statistical method used (e.g., MantelHaenszel or Peto);

Results of sensitivity analyses applied to account for MPD.
Phase 2: we will abstract the following data from all trials included in each eligible metaanalysis:
Characteristics of the trials:

Type of reference (e.g., abstract, fulltext article);

Type of trial (e.g., randomized controlled trial, controlled clinical trial);

Duration of followup for outcome measurement;

Source of funding (e.g., private for profit, private not for profit, governmental, not funded, not reported);

Type of outcome (e.g., efficacy, safety).
Missing participant data:

Presence of MPD;

Whether the trialists report the MPD for this specific outcome, as opposed to reporting premature end of followup for trial participants in general;

Whether the trialists followedup categories of participants that could be potentially counted as having MPD (see previous section “Categories of participants with potential MPD”);

Whether the trialists included in the analysis the categories of participants that could be potentially counted as having MPD in the analysis;

Level of reporting number of participants with MPD (e.g., per arm, both arms combined);

Whether the trialists described the pattern of missingness;

Whether the trialists compared the baseline characteristics of participants with and without MPD (e.g., MPD group vs. nonMPD group, MPD first arm vs. MPD second arm)

Number of participants in each of the categories of participants that, following our rules stated previously, would be counted as having MPD, per arm;

Whether the trialists report using the following for their analysis: intentiontotreat, modified intentiontotreat, per protocol, or as treated;

Whether the trialists report analyzing participants in the group to which they were randomized;

Whether the trialists report the analytical method for dealing with MPD in the main analysis, i.e., to account for MPD when generating the effect estimate (e.g., complete case analysis, making assumptions for MPD);

Whether methods using assumptions for MPD accounted for uncertainty associated with imputing events;

Whether the trialists provided any justification for the method used to deal with MPD in the main analysis;

Method(s) used to assess risk of bias associated with MPD in the analysis (e.g., sensitivity analysis, different methods for different subgroups of participants with MPD), if any;

Numerical information for the comparison and outcome of interest: number randomized per arm, number of observed events per arm (see Table 1);

Relative effect measure (e.g., RR, OR) and its associated CI, and p value;

Analytical method “actually” used by the systematic review authors based on the numerical information provided by the trialists;

Whether the trialists report any additional analytical methods for dealing with MPD (“reported analytical method”), i.e., to judge the risk of bias associated with MPD (e.g., complete case analysis, making assumptions for MPD);

Whether the trialists incorporated implications of MPD in results or discussion sections.
Sample size
The sample size will be based on the goal of achieving enough precision (CI width of 0.1) around the proportion of systematic reviews (both Cochrane and nonCochrane) for which the pooled effect estimate crosses the boundary of conventional statistical significance (further details below under “Impact of different methods of dealing with MPD on pooled effect estimates”). We will use the findings after including the first 100 systematic reviews to calculate the final sample size. We will include equal numbers of Cochrane and nonCochrane systematic reviews.
Data analysis
Agreement
We will assess agreement between reviewers for inclusion of systematic reviews at the fulltext screening stage for their first judgment (i.e., before the step of reaching agreement by consensus) using chancecorrected agreement (kappa statistic). If fewer than 15% or more than 85% of citations are included in this study, we will measure agreement using chanceindependent agreement (phi statistic)[10]. We will interpret the agreement statistics using the guidelines proposed by Landis and Koch[11]: kappa values of 0 to 0.20 represent slight agreement, 0.21 to 0.40 fair agreement, 0.41 to 0.60 moderate agreement, 0.61 to 0.80 substantial agreement, and greater than 0.80 almost perfect agreement. We will use these same thresholds for interpreting phi.
Characteristics of the included systematic reviews
We will conduct a descriptive analysis of the characteristics of included systematic reviews and the comparisons of interest. For all descriptive analyses, we will use percentages for dichotomous variables. To describe the distribution of continuous variables, we will use the mean and standard deviation when distribution is near normal and median and interquartile range (IQR) when the distribution is substantially skewed. We will use the ShapiroWilk test to evaluate whether the distributions of continuous variables violate assumptions of normality.
For all relevant analyses, we will test for the statistical significance of the differences between the Cochrane and nonCochrane reviews. For dichotomous variables, we will use the chisquare test. For continuous variables, we will use Student’s two independent sample ttest for two independent samples when the distribution is normal and the Mann–Whitney Utest (a nonparametric test for two independent samples) when the distribution is not normal. When results differ, we will present the results stratified by the systematic review type (i.e., Cochrane vs. nonCochrane) and consider presenting the overall results as well. When results do not differ, we will focus on the overall results but consider presentation of stratified results as well.
Categories of trial participants reviewers considered as having MPD
For each category of participants that could be potentially counted as having MPD (refer to section above on “Categories of participants with potential MPD”), we will calculate:

The percentage of systematic reviews documenting that category;

The percentage of trials documenting that category;

The distribution across trials of the proportion of participants belonging to that category.
We will compare (1) data from the trial regarding the number of participants in the different categories of participants that, following our rules stated previously, would be counted as having MPD, to (2) the number of participants considered in the systematic review as having MPD. We will conduct this analysis irrespective of what categories of participants that could be potentially counted as having MPD are reported by the systematic review.
Reporting of MPD by trialists
We will conduct a descriptive analysis of the reporting of MPD by trialists. We will calculate:

The percentage of trials followingup categories of participants that could be potentially counted as having MPD;

The percentage of trials documenting the number of participants in each arm with MPD; if applicable;

The percentage of trials documenting different numbers of participants with MPD across different outcomes, if applicable;

The percentage of trials documenting the pattern of missingness;

The percentage of trials documenting the comparison of the baseline characteristics of participants with and without missing participant data;

The percentage of trials documenting the methods used to assess risk of bias associated with MPD in the analysis.
Reported vs. actual method of dealing with of MPD in metaanalysis
First, we will conduct the following descriptive analysis:

The percentage of systematic reviews using each of the terms “intentiontotreat,” “modified intentiontotreat,” “per protocol,” and “as treated;”

The percentage of systematic reviews reporting a plan to analyze participants in the group to which they were randomized;

The percentage of systematic reviews using different analytical methods for dealing with MPD in the main analysis, i.e., when generating the best effect estimate;

The percentage of systematic reviews using imputed data for MPD and accounting for uncertainty associated with imputing events;

The percentage of systematic reviews justifying the analytical method used to deal with MPD in the eligible metaanalysis;

The percentage of systematic reviews using different analytical methods (e.g., sensitivity analysis, subgroup analysis) to judge risk of bias associated with MPD;

The percentage of systematic reviews using different tools (e.g., Cochrane RoB tool) to measure risk of bias associated with MPD in individual trials, and for the body of evidence, across trials, supporting the outcome of interest.
Second, we will analyze the “actually used analytical method” for dealing with MPD. For this purpose, and for each trial included in the eligible metaanalysis, we will compare data from the trial report (number randomized, number of observed events, number with missing data) to the corresponding data included in the metaanalysis (numerator, denominator). Based on these comparisons, we will categorize the “actually used analytical method” for dealing with MPD as: complete case analysis, making assumptions for MPD, using the assumption used by the trialists, or excluding trials with high rate of MPD.
We will develop and pilot test specific rules for categorizing the “actually used analytical method” for dealing with MPD. As part of this process, we will note when some of these data appear to have been wrongly abstracted. Two reviewers will make these judgments in duplicate and independently. They will resolve discrepancies by consensus, and if unsuccessful, with the help of a third reviewer.
Third, we will conduct a bivariate analysis to assess the association between the “actually used analytical method” and the following characteristics:

Whether the trialists report using the following for their analysis: intentiontotreat, modified intentiontotreat, per protocol, as treated;

Whether the trialists report analyzing participants in the group to which they were randomized.
Fourth, we will compare the “actually used analytical method” for dealing with MPD with the “reported analytical method.” The latter refers to the method the authors report for dealing with MDP (e.g., complete case analysis, making assumptions for MPD, using the assumption used by the trialists, or excluding trials with high rate of MPD). We will calculate the percentage of systematic reviews with “discrepancy” between “reported” and “actually used” analytical methods. We will display the results in a matrix with “reported” analytical methods displayed in columns and “actually used” analytical methods displayed in rows. We will test whether the distribution of studies according to these two variables is due to chance using the chisquare statistical test.
Finally, we will conduct a multivariable logistic regression analysis using the “discrepancy” as the dependent variable and the general characteristics of the included systematic reviews as the independent variables: type of systematic review (i.e., Cochrane vs. nonCochrane), use of GRADE, source of funding, type of intervention, and outcome patientimportance ranking.
Impact of different methods of dealing with MPD on pooled effect estimates
For the following analyses, we will use “original effect estimate” to refer to the effect estimate reported in the eligible metaanalyses. For each eligible metaanalysis, we will first attempt to reproduce the original analysis. When this analysis generates a different effect estimate that is not statistically significant, we will exclude the corresponding metaanalysis from this part of the study.
Then, we will apply a number of methods of dealing with MPD (see below). Each of these methods will generate a set of values for the numerator and denominator in each arm of each trial included in the metaanalysis. We will use these values to conduct the sensitivity analyses using the same pooled relative effect measure (RR or OR), the same analysis model (random effect or fixed effect), and the same statistical method (e.g., MantelHaenszel, Peto) used by the systematic review authors to generate the original effect estimate. We will use the “assumption pooled effect estimates” to refer to the results of those sensitivity analyses. For outcomes that have a measure of effect greater than 1.00, we will reverse the direction of the comparison at the time of the analysis.
In terms of applying different methods of dealing with MPD, we will start by conducting a complete case analysis. As described in detail in the subsequent paragraphs, we will also apply both “implausible” (but sometimes used) and more plausible assumptions for dealing with MPD. We will apply to each of these assumptions and statistical approaches to take uncertainty into account.
We will use the following implausible assumptions,[5] described in more details in Table 2:

None of the participants with MPD had the event;

All participants with MPD had the event;

None of the participants with MPD in the treatment group had the event and all participants with MPD in the control group did (best case scenario);

All participants with MPD in the treatment group had the event and none of participants with MPD in the control group did (worst case scenario).
We will also use the more plausible assumptions that the event incidence among participants with missing data is equal to or higher than the observed event incidence among followedup and by a specified ratio (see Table 2). For this purpose, we define RI_{NotFU/FU} as the relative event incidence among those not followedup relative to the event incidence among those followedup. We will test assumptions combining a range of plausible RI_{NotFU/FU} values (1, 1.5, 2, 3, and 5) in the two comparison arms. This will follow the methodology our group used to conduct a similar project focusing on trials (the LOSTIT project)[8]. In addition, we will use the assumption that the incidence for missing participants in both arms is the same as the one observed in the trial control arm.
Making assumptions implies imputing the outcomes of participants with missing data. That would increase the total number of events and result in inappropriate narrowing the CI of the effect estimate. In order to avoid such inappropriate narrowing of the CI, we will apply statistical approaches to take into account the uncertainty associated with imputing outcomes. One recommended approach consists of using a “variance inflation factor” to inflate the standard errors or variances of effect estimates[12, 13].
For each of the assumptions, we will calculate the proportion of systematic reviews for which the “assumption pooled effect estimate” crosses the boundary of conventional statistical significance.
Discussion
The main objective of this study is to assess in Cochrane and nonCochrane systematic reviews the impact of different methods of dealing with MPD on pooled effect estimates of dichotomous outcomes, with focus on systematic reviews reporting statistically significant findings. We will also compare the number of participants that following our rules would be counted as having MPD with the number of participants considered in the systematic review counted as having MPD.
The strengths of our proposed study include the adoption of systematic and transparent methods, including specific and explicit eligibility criteria, sensitive search strategies, and duplicate and independent processes for study selection, data abstraction, and data interpretation (e.g., judgment about the “actually used” analytical methods). Also, we will use standardized and pilot tested forms supplemented with specific and detailed instructions. We will explore issues that have not previously been addressed, particularly in terms of the impact of MPD on pooled effect estimates. Finally, our group has extensive experience in completing methodological studies involving large samples[1, 14–18].
The major limitation of our study is the need for reviewers’ judgments at different stages of the process (e.g., judgment about the “actually used” analytical methods). The specific and detailed instructions pilot testing, and calibration exercises described previously should help minimize disagreement.
This study will focus on dichotomous outcome data, given the methodological and statistical issues vary substantively for continuous data. Also, it will focus on systematic reviews with a statistically significant pooled effect estimate for a patientimportant efficacy outcome because they are the most likely to influence clinical practice.
We will be assessing how Cochrane and nonCochrane systematic reviews compare across the three objectives of this study. A major reason is that the former reviews tend to be of higher methodological quality compared to the latter[19, 20]. Indeed, we opted to assess the quality of the included systematic reviews through using two indicators: the type of systematic review (Cochrane vs. nonCochrane) and the use of the GRADE approach. In a recently conducted methodological survey on MPD in systematic review, regression analyses identified these two factors, but not the AMSTAR tool,[11] as statistically significant predictors of two MPD related variables (i.e., whether the systematic review judged the risk of bias associated with MPD and whether the systematic review used complete case analysis) (unpublished data).
For the third objective of this study, and in addition to assessing the impact of plausible methods of dealing with MPD, we will assess the impact of implausible methods. This is because these implausible methods (e.g., none had the event, all had the event) are sometimes used in published metaanalyses. Also, the worst case scenario could be used to test the robustness of the results[5].
Understanding which categories of trial participants the systematic review authors consider as having MPD and how they deal with MPD in metaanalysis will help in better defining standards for conducting and reporting systematic reviews. Assessing the impact of different methods of dealing with MPD on pooled effect estimates will help judge the associated risk of bias in systematic reviews. Our findings will inform recommendations regarding what assumptions for MPD should be used to test the robustness of the metaanalytical results. They will also help in better defining how to assess risk of bias associated with MPD at the systematic review level and provide insight into the frequency with which MPD compromises trust in statistically significant metaanalytical results. Finally, the publication of this protocol will contribute to making the objectives and design of methodological surveys more transparent[2, 8, 21–24].
Abbreviations
 MPD:

missing participant data
 RoB:

risk of bias
 RR:

relative risk
 OR:

odds ratio
 CI:

confidence interval
 RCT:

randomized controlled trial
 CCT:

controlled clinical trial
 IQR:

interquartile range
 RI_{NotFU/FU} :

the relative event incidence among those not followedup relative to the event incidence among those followedup
 AMSTAR:

a measurement tool to assess systematic reviews.
Declarations
Acknowledgments
We would like to acknowledge Neera Bhatnagar for her help with developing the search strategy and Dr. Reem Waziry for her help with pilot testing the screening forms. This research was supported by funds provided by The Cochrane Collaboration's Methods Innovation Fund. The views expressed in this final report are those of the authors and not necessarily those of The Cochrane Collaboration or its registered entities, committees, or working groups.
Authors’ Affiliations
References
 Akl EA, Briel M, You JJ, Sun X, Johnston BC, Busse JW, Mulla S, Lamontagne F, Bassler D, Vera C, Alshurafa M, Katsios CM, Zhou Q, CukiermanYaffe T, Gangji A, Mills EJ, Walter SD, Cook DJ, Schunemann HJ, Altman DG, Guyatt GH: Potential impact on estimated treatment effects of information lost to followup in randomised controlled trials (LOSTIT): systematic review. BMJ. 2012, 344: e280910.1136/bmj.e2809.View ArticlePubMedGoogle Scholar
 AlonsoCoello P, CarrascoLabra A, BrignardelloPetersen R, Neumann I, Akl EA, Sun X, Johnston BC, Briel M, Busse JW, Glujovsky D, Granados CE, Iorio A, Irfan A, Garcia LM, Mustafa RA, RamirezMorera A, Sola I, Tikkinen KA, Ebrahim S, Vandvik PO, Zhang Y, Selva A, Sanabria AJ, Zazueta OE, Vernooij RW, Schunemann HJ, Guyatt GH: A methodological survey of the analysis, reporting and interpretation of absolute risk reduction in systematic revieWs (ARROW): a study protocol. Syst Rev. 2013, 2: 11310.1186/204640532113.View ArticlePubMedPubMed CentralGoogle Scholar
 Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA, Cochrane Bias Methods G, Cochrane Statistical Methods G: The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011, 343: d592810.1136/bmj.d5928.View ArticlePubMedPubMed CentralGoogle Scholar
 Savovic J, Weeks L, Sterne JA, Turner L, Altman DG, Moher D, Higgins JP: Evaluation of the Cochrane collaboration’s tool for assessing the risk of bias in randomized trials: focus groups, online survey, proposed recommendations and their implementation. Syst Rev. 2014, 3: 3710.1186/20464053337.View ArticlePubMedPubMed CentralGoogle Scholar
 Akl EA, Johnston BC, AlonsoCoello P, Neumann I, Ebrahim S, Briel M, Cook DJ, Guyatt GH: Addressing dichotomous data for participants excluded from trial analysis: a guide for systematic reviewers. PLoS One. 2013, 8: e5713210.1371/journal.pone.0057132.View ArticlePubMedPubMed CentralGoogle Scholar
 Ebrahim S, Akl EA, Mustafa RA, Sun X, Walter SD, HeelsAnsdell D, AlonsoCoello P, Johnston BC, Guyatt GH: Addressing continuous data for participants excluded from trial analysis: a guide for systematic reviewers. Clin Epidemiol. 2013, 66: 10141021. e1011. 10.1016/j.jclinepi.2013.03.014.View ArticleGoogle Scholar
 Ebrahim S, Johnston BC, Akl EA, Mustafa RA, Sun X, Walter SD, HeelsAnsdell D, AlonsoCoello P, Guyatt GH: Addressing continuous data measured with different instruments for participants excluded from trial analysis: a guide for systematic reviewers. J Clin Epidemiol. 2014, 67: 560570. 10.1016/j.jclinepi.2013.11.014.View ArticlePubMedGoogle Scholar
 Akl EA, Briel M, You JJ, Lamontagne F, Gangji A, CukiermanYaffe T, Alshurafa M, Sun X, Nerenberg KA, Johnston BC, Vera C, Mills EJ, Bassler D, Salazar A, Bhatnagar N, Busse JW, Khalid Z, Walter S, Cook DJ, Schunemann HJ, Altman DG, Guyatt GH: LOST to followup information in trials (LOSTIT): a protocol on the potential impact. Trials. 2009, 10: 4010.1186/174562151040.View ArticlePubMedPubMed CentralGoogle Scholar
 Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, FalckYtter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schunemann HJ: GRADE guidelines: 1. IntroductionGRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011, 64: 383394. 10.1016/j.jclinepi.2010.04.026.View ArticlePubMedGoogle Scholar
 Cramer H: Mathematical Methods of Statistics. 1946, Princeton: Princeton University PressGoogle Scholar
 Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007, 7: 1010.1186/14712288710.View ArticlePubMedPubMed CentralGoogle Scholar
 White IR, Higgins JP, Wood AM: Allowing for uncertainty due to missing data in metaanalysis—part 1: twostage methods. Stat Med. 2008, 27: 711727. 10.1002/sim.3008.View ArticlePubMedGoogle Scholar
 White IR, Welton NJ, Wood AM, Ades AE, Higgins JP: Allowing for uncertainty due to missing data in metaanalysis—part 2: hierarchical models. Book Allowing for Uncertainty due to Missing Data in MetaAnalysis  Part 2: Hierarchical Models, Volume 27. 2008, 728745.Google Scholar
 FerreiraGonzalez I, Busse JW, HeelsAnsdell D, Montori VM, Akl EA, Bryant DM, AlonsoCoello P, Alonso J, Worster A, Upadhye S, Jaeschke R, Schunemann HJ, PermanyerMiralda G, PachecoHuergo V, DomingoSalvany A, Wu P, Mills EJ, Guyatt GH: Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials. BMJ. 2007, 334: 78610.1136/bmj.39136.682083.AE.View ArticlePubMedPubMed CentralGoogle Scholar
 Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, DiazGranados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, AlonsoCoello P, Hassouneh B, Truong J, Dattani ND, Walter SD, HeelsAnsdell D, Bhatnagar N, Altman DG, Guyatt GH: The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review. BMJ. 2011, 342: d156910.1136/bmj.d1569.View ArticlePubMedGoogle Scholar
 Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, DiazGranados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, AlonsoCoello P, Hassouneh B, Walter SD, HeelsAnsdell D, Bhatnagar N, Altman DG, Guyatt GH: Credibility of claims of subgroup effects in randomised controlled trials: systematic review. BMJ. 2012, 344: e155310.1136/bmj.e1553.View ArticlePubMedGoogle Scholar
 Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, HeelsAnsdell D, Walter SD, Guyatt GH, Flynn DN, Elamin MB, Murad MH, Abu Elnour NO, Lampropulos JF, Sood A, Mullan RJ, Erwin PJ, Bankhead CR, Perera R, Ruiz Culebro C, You JJ, Mulla SM, Kaur J, Nerenberg KA, Schunemann H, Cook DJ, Lutz K, Ribic CM, Vale N, StopitStudy Group: Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and metaregression analysis. JAMA. 2010, 303: 11801187. 10.1001/jama.2010.310.View ArticlePubMedGoogle Scholar
 Kasenda B, von Elm E, You J, Blumle A, Tomonaga Y, Saccilotto R, Amstutz A, Bengough T, Meerpohl JJ, Stegert M, Tikkinen KA, Neumann I, CarrascoLabra A, Faulhaber M, Mulla SM, Mertz D, Akl EA, Bassler D, Busse JW, FerreiraGonzalez I, Lamontagne F, Nordmann A, Gloy V, Raatz H, Moja L, Rosenthal R, Ebrahim S, Schandelmaier S, Xin S, Vandvik PO: Prevalence, characteristics, and publication of discontinued randomized trials. JAMA. 2014, 311: 10451051. 10.1001/jama.2014.1361.View ArticlePubMedGoogle Scholar
 Hopewell S, Boutron I, Altman DG, Ravaud P: Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials: a crosssectional study. BMJ Open. 2013, 3: e003342View ArticlePubMedPubMed CentralGoogle Scholar
 Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C: Cochrane reviews used more rigorous methods than noncochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009, 62: 10211030. 10.1016/j.jclinepi.2008.09.018.View ArticlePubMedGoogle Scholar
 Briel M, Lane M, Montori VM, Bassler D, Glasziou P, Malaga G, Akl EA, FerreiraGonzalez I, AlonsoCoello P, Urrutia G, Kunz R, Culebro CR, da Silva SA, Flynn DN, Elamin MB, Strahm B, Murad MH, Djulbegovic B, Adhikari NK, Mills EJ, GwadrySridhar F, Kirpalani H, Soares HP, Abu Elnour NO, You JJ, Karanicolas PJ, Bucher HC, Lampropulos JF, Nordmann AJ, Burns KE: Stopping randomized trials early for benefit: a protocol of the study of trial policy of interim truncation2 (STOPIT2). Trials. 2009, 10: 4910.1186/174562151049.View ArticlePubMedPubMed CentralGoogle Scholar
 Kasenda B, von Elm EB, You J, Blumle A, Tomonaga Y, Saccilotto R, Amstutz A, Bengough T, Meerpohl J, Stegert M, Tikkinen KA, Neumann I, CarrascoLabra A, Faulhaber M, Mulla S, Mertz D, Akl EA, Bassler D, Busse JW, FerreiraGonzalez I, Lamontagne F, Nordmann A, Rosenthal R, Schandelmaier S, Sun X, Vandvik PO, Johnston BC, Walter MA, Burnand B, Schwenkglenks M: Learning from failure–rationale and design for a study about discontinuation of randomized trials (DISCO study). BMC Med Res Methodol. 2012, 12: 13110.1186/1471228812131.View ArticlePubMedPubMed CentralGoogle Scholar
 Sun X, Briel M, Busse JW, Akl EA, You JJ, Mejza F, Bala M, DiazGranados N, Bassler D, Mertz D, Srinathan SK, Vandvik PO, Malaga G, Alshurafa M, Dahm P, AlonsoCoello P, HeelsAnsdell DM, Bhatnagar N, Johnston BC, Wang L, Walter SD, Altman DG, Guyatt GH: Subgroup analysis of trials is rarely easy (SATIRE): a study protocol for a systematic review to characterize the analysis, reporting, and claim of subgroup effects in randomized trials. Trials. 2009, 10: 10110.1186/1745621510101.View ArticlePubMedPubMed CentralGoogle Scholar
 Wu D, Akl EA, Guyatt GH, Devereaux PJ, BrignardelloPetersen R, Prediger B, Patel K, Patel N, Lu T, Zhang Y, Falavigna M, Santesso N, Mustafa RA, Zhou Q, Briel M, Schunemann HJ: Methodological survey of designed uneven randomization trials (DURANDOM): a protocol. Trials. 2014, 15: 3310.1186/174562151533.View ArticlePubMedPubMed CentralGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.