A protocol for a systematic review on the impact of unpublished studies and studies published in the gray literature in meta-analyses

Background Meta-analyses are particularly vulnerable to the effects of publication bias. Despite methodologists’ best efforts to locate all evidence for a given topic the most comprehensive searches are likely to miss unpublished studies and studies that are published in the gray literature only. If the results of the missing studies differ systematically from the published ones, a meta-analysis will be biased with an inaccurate assessment of the intervention’s effects. As part of the OPEN project (http://www.open-project.eu) we will conduct a systematic review with the following objectives: ▪ To assess the impact of studies that are not published or published in the gray literature on pooled effect estimates in meta-analyses (quantitative measure). ▪ To assess whether the inclusion of unpublished studies or studies published in the gray literature leads to different conclusions in meta-analyses (qualitative measure). Methods/Design Inclusion criteria: Methodological research projects of a cohort of meta-analyses which compare the effect of the inclusion or exclusion of unpublished studies or studies published in the gray literature. Literature search: To identify relevant research projects we will conduct electronic searches in Medline, Embase and The Cochrane Library; check reference lists; and contact experts. Outcomes: 1) The extent to which the effect estimate in a meta-analyses changes with the inclusion or exclusion of studies that were not published or published in the gray literature; and 2) the extent to which the inclusion of unpublished studies impacts the meta-analyses’ conclusions. Data collection: Information will be collected on the area of health care; the number of meta-analyses included in the methodological research project; the number of studies included in the meta-analyses; the number of study participants; the number and type of unpublished studies; studies published in the gray literature and published studies; the sources used to retrieve studies that are unpublished, published in the gray literature, or commercially published; and the validity of the methodological research project. Data synthesis: Data synthesis will involve descriptive and statistical summaries of the findings of the included methodological research projects. Discussion Results are expected to be publicly available in the middle of 2013.


Background
A meta-analyses as part of a systematic review aims to provide a thorough, comprehensive and unbiased account of the literature [1,2]. However, potentially important studies could be missing from a meta-analysis because of selective publication and inadequate dissemination of results. Despite methodologists' best efforts to locate all eligible evidence for a given topic the most comprehensive searches are likely to miss unpublished studies and studies that are not commercially published and, therefore, are not indexed in respective databases (so called gray literature, such as conference abstract, dissertations, policy documents, book chapters). If the results from missing studies differ systematically from the published data, a metaanalysis may become biased with an inaccurate assessment of the intervention's effects. For instance, positive, significant findings are more likely to be published than non-significant findings, and a metaanalysis which is based mainly on published literature may end up overestimating the efficacy of the intervention [3][4][5].
However, the impact of gray literature and unpublished studies on the conclusions of meta-analyses has not been comprehensively clarified. For example, there is some evidence that suggests that published randomized controlled trials (RCTs) tend to be larger and show an overall greater treatment effect than gray trials [6]. But the identification of relevant unpublished studies or studies published in the gray literature and their inclusion in meta-analyses can be particularly timeconsuming and challenging. There is also some controversy as to whether unpublished studies and studies published in the gray literature should be included in meta-analyses because they might be incomplete and their methodological quality (validity) can be difficult to assess. A publication by Cook and colleagues in 1993 showed that 78% of authors of meta-analyses felt that unpublished studies should be included in meta-analyses compared to only 47% of journal editors [7]. Therefore, research is needed to help assess the potential implications for reviewers of not including gray literature and unpublished studies in meta-analyses of health care interventions.

Objectives
In terms of the above mentioned controversies regarding the inclusion of unpublished studies and studies published in the gray literature on the results of metaanalyses, we will conduct a systematic review with the following objectives: ▪ To assess the impact of studies that were not published or only published in the gray literature on pooled effect estimates in meta-analyses (quantitative measure) ▪ To assess whether the inclusion of unpublished studies or studies published in the gray literature impacts the conclusions of meta-analyses (qualitative measure) This systematic review will be part of the OPEN Project (To Overcome failure to Publish nEgative fiNdings) which was developed with the goal of elucidating the scope of non-publication of studies through a series of systematic reviews. In an earlier issue of this journal ('Systematic Reviews'), our group has already published a protocol for a systematic review which evaluates the extent of non-publication of research studies, which were approved by ethics committees, registered in trial registries or presented as conference abstracts [8].

Search methods for identification of methodological research projects
To identify the relevant research evidence we will conduct electronic literature searches in the following databases: Ovid Medline (1946 to present), Ovid Medline Daily Update, Ovid Medline in process & other nonindexed citations, Ovid Embase (1980 to present), The Cochrane Library (most current issue) and Web of Science. No language restrictions will be applied.
In addition, the bibliographies of any eligible articles identified will be checked for additional references and citation searches will be carried out for all included references using ISI Web of Knowledge.
A search strategy for the electronic literature search in Ovid Medline has already been designed with the support of a librarian/information specialist. This strategy was translated as appropriate for the other databases (for the full search strategies see Appendix A). In addition, we will contact various experts in the field for further eligible studies.

Data collection and analyses Selection of methodological research projects
A methodological research project will be considered eligible for inclusion in this systematic review if it reviews a cohort of meta-analyses (that is, more than one metaanalyses) that: ▪ compare pooled effect estimates of meta-analyses of health care interventions according to publication status (that is, published versus unpublished studies or gray literature) or ▪ examine whether the inclusion of unpublished studies or gray literature impacts the overall findings or conclusions of a meta-analyses We will consider 'published' articles to be manuscripts that appeared in peer-reviewed journals. Our working definition of gray literature will correspond to the definitions used by the authors of eligible methodological research projects and which also conforms to the definition of 'gray literature' described earlier in this protocol (see Background). A meta-analysis is defined as the calculation of a summary estimate of treatment effect by pooling the results of two or more studies.

Data extraction
A specifically designed data extraction form will be developed and two reviewers will independently extract all relevant data from eligible methodological research projects. The following information will be collected:

Assessment of validity
We will systematically consider the validity and generalizability of the identified evidence provided by each of the methodological research projects by evaluating the following aspects: Significance and clinical relevance will be defined according to the definitions provided in the methodological research project.

Unit of analyses issues
The anticipated unit of analyses is the meta-analyses included in the methodological research project.

Assessment of heterogeneity
Heterogeneity for pooled outcome measures will be assessed by standard methods including Chi 2 -test and calculation of the I 2 value [9].

Assessment of reporting biases
Funnel plots will be used to assess the association between point estimates of log odds ratio (a measure of extent of association between meta-analyses' characteristics and change in summary estimates) and a measure of precision if more than ten methodological research projects provide necessary information. Funnel plots will be visually assessed and appropriate formal statistical tests following recommendations formulated by Sterne et al. will be used to test for asymmetry [5]. In the instance of suspected reporting bias authors will be contacted.

Data synthesis
Data synthesis will involve a combination of descriptive and statistical summaries of the impact of the inclusion or exclusion of unpublished studies and gray literature on the results of meta-analyses (identified by methodological research projects).
The decision on whether or not to combine the results of the included methodological research projects will depend on the assessment of heterogeneity. Where methodological research projects will be judged to be sufficiently homogenous in their design a meta-analyses of these research projects will be carried out. The estimated ratios of unpublished and published in the gray literature only versus published treatment effects generated from each methodological evaluation will then be used to summarize the overall difference in risk ratios between unpublished and published in the gray literature only and published studies. The 95% confidence interval for the combined effect will be estimated using a random effects model.

Subgroup analyses and investigation of heterogeneity
The following subgroup analyses are planned:

Sensitivity analyses
No sensitivity analyses are planned. However, should the instance arise where a methodological research project with doubtful eligibility is identified (as determined by individual review and analyses of the validity of the methodological research project), sensitivity analyses may be undertaken. Possible sensitivity analyses may be based on the following: ▪ Methodological quality/validity of the methodological research project; only methodological research projects with low risk of bias will be considered. A methodological research project will be of high risk of bias when the external validity and/or the validity of the included metaanalyses are doubtful.

Discussion
This systematic review seeks to comprehensively synthesize the growing body of research that is related to the impact of including unpublished studies and studies published in the gray literature in meta-analyses. By considering multiple characteristics and potential confounders related to unpublished studies and studies published in the gray literature, we hope to identify sufficient evidence to conclude whether (or to what extent) inclusion of unpublished studies and studies published in the gray literature has an impact on the pooled effect estimates and the conclusions from a meta-analyses. The findings, including risk factors for unpublished studies and studies published in the gray literature, will have important implications for researchers conducting metaanalyses since they need to be informed about the impact and extent of (not) including unpublished and gray studies in meta-analyses. In addition, this systematic review in combination with the results of other systematic reviews that are part of the OPEN Project will serve to raise awareness about the impact of publication bias and the complexity of this issue. These reviews will also serve as a foundation for a recommendations workshop which will enable key members of the biomedical research community (for example, funders, research ethics committees, and journal editors) to develop future policies and guidelines to lessen the frequency of nonpublication and related biases. We acknowledge that more than half of all systematic reviews do not involve meta-analysis in their analyses. Despite the fact that our main outcomes focus on the impact of unpublished and gray studies on pooled effect estimates in meta-analyses, our findings will also be valuable for systematic reviews. It is obvious, if we find a statistical difference in the pooled effect estimates in meta-analyses, it is also likely that gray and unpublished literature impacts descriptive results of systematic reviews. Beside effect estimates, we will also evaluate differences in the methodological quality and study characteristics (such as number of participants, language or methodological quality) between unpublished, gray and published studies. These results will also be valuable for systematic reviews to appraise the potential impact of publication bias.

Appendix A
Search Strategy for OvidSP MEDLINE (search strategy will be adapted for other databases) Line 10 to 14 and 16 to 35 are not shown because they are part of the search strategy for our 1 st systematic review [8] and, therefore, have no consequence for this search.