A protocol for a systematic review on the impact of unpublished studies and studies published in the gray literature in meta-analyses

  • Christine Schmucker1,

    Affiliated with

    • Annette Bluemle1,

      Affiliated with

      • Matthias Briel2, 3,

        Affiliated with

        • Susan Portalupi1,

          Affiliated with

          • Britta Lang1,

            Affiliated with

            • Edith Motschall4,

              Affiliated with

              • Guido Schwarzer4,

                Affiliated with

                • Dirk Bassler5,

                  Affiliated with

                  • Katharina F Mueller5,

                    Affiliated with

                    • Erik von Elm1, 6,

                      Affiliated with

                      • Joerg J Meerpohl1Email author and

                        Affiliated with

                        • on behalf of the OPEN Consortium

                          Affiliated with

                          Systematic Reviews20132:24

                          DOI: 10.1186/2046-4053-2-24

                          Received: 23 January 2013

                          Accepted: 15 April 2013

                          Published: 2 May 2013

                          Abstract

                          Background

                          Meta-analyses are particularly vulnerable to the effects of publication bias. Despite methodologists’ best efforts to locate all evidence for a given topic the most comprehensive searches are likely to miss unpublished studies and studies that are published in the gray literature only. If the results of the missing studies differ systematically from the published ones, a meta-analysis will be biased with an inaccurate assessment of the intervention’s effects.

                          As part of the OPEN project (http://​www.​open-project.​eu) we will conduct a systematic review with the following objectives:
                          • ▪ To assess the impact of studies that are not published or published in the gray literature on pooled effect estimates in meta-analyses (quantitative measure).

                          • ▪ To assess whether the inclusion of unpublished studies or studies published in the gray literature leads to different conclusions in meta-analyses (qualitative measure).

                          Methods/Design

                          Inclusion criteria: Methodological research projects of a cohort of meta-analyses which compare the effect of the inclusion or exclusion of unpublished studies or studies published in the gray literature.

                          Literature search: To identify relevant research projects we will conduct electronic searches in Medline, Embase and The Cochrane Library; check reference lists; and contact experts.

                          Outcomes: 1) The extent to which the effect estimate in a meta-analyses changes with the inclusion or exclusion of studies that were not published or published in the gray literature; and 2) the extent to which the inclusion of unpublished studies impacts the meta-analyses’ conclusions.

                          Data collection: Information will be collected on the area of health care; the number of meta-analyses included in the methodological research project; the number of studies included in the meta-analyses; the number of study participants; the number and type of unpublished studies; studies published in the gray literature and published studies; the sources used to retrieve studies that are unpublished, published in the gray literature, or commercially published; and the validity of the methodological research project.

                          Data synthesis: Data synthesis will involve descriptive and statistical summaries of the findings of the included methodological research projects.

                          Discussion

                          Results are expected to be publicly available in the middle of 2013.

                          Keywords

                          Publication bias Gray literature Unpublished studies Meta-analyses The OPEN project

                          Background

                          A meta-analyses as part of a systematic review aims to provide a thorough, comprehensive and unbiased account of the literature [1, 2]. However, potentially important studies could be missing from a meta-analysis because of selective publication and inadequate dissemination of results. Despite methodologists’ best efforts to locate all eligible evidence for a given topic the most comprehensive searches are likely to miss unpublished studies and studies that are not commercially published and, therefore, are not indexed in respective databases (so called gray literature, such as conference abstract, dissertations, policy documents, book chapters). If the results from missing studies differ systematically from the published data, a meta-analysis may become biased with an inaccurate assessment of the intervention’s effects. For instance, positive, significant findings are more likely to be published than non-significant findings, and a meta-analysis which is based mainly on published literature may end up overestimating the efficacy of the intervention [35].

                          However, the impact of gray literature and unpublished studies on the conclusions of meta-analyses has not been comprehensively clarified. For example, there is some evidence that suggests that published randomized controlled trials (RCTs) tend to be larger and show an overall greater treatment effect than gray trials [6]. But the identification of relevant unpublished studies or studies published in the gray literature and their inclusion in meta-analyses can be particularly time-consuming and challenging. There is also some controversy as to whether unpublished studies and studies published in the gray literature should be included in meta-analyses because they might be incomplete and their methodological quality (validity) can be difficult to assess. A publication by Cook and colleagues in 1993 showed that 78% of authors of meta-analyses felt that unpublished studies should be included in meta-analyses compared to only 47% of journal editors [7]. Therefore, research is needed to help assess the potential implications for reviewers of not including gray literature and unpublished studies in meta-analyses of health care interventions.

                          Objectives

                          In terms of the above mentioned controversies regarding the inclusion of unpublished studies and studies published in the gray literature on the results of meta-analyses, we will conduct a systematic review with the following objectives:
                          • ▪ To assess the impact of studies that were not published or only published in the gray literature on pooled effect estimates in meta-analyses (quantitative measure)

                          • ▪ To assess whether the inclusion of unpublished studies or studies published in the gray literature impacts the conclusions of meta-analyses (qualitative measure)

                          This systematic review will be part of the OPEN Project (To Overcome failure to Publish nEgative fiNdings) which was developed with the goal of elucidating the scope of non-publication of studies through a series of systematic reviews. In an earlier issue of this journal (‘Systematic Reviews’), our group has already published a protocol for a systematic review which evaluates the extent of non-publication of research studies, which were approved by ethics committees, registered in trial registries or presented as conference abstracts [8].

                          Methods/Design

                          Search methods for identification of methodological research projects

                          To identify the relevant research evidence we will conduct electronic literature searches in the following databases: Ovid Medline (1946 to present), Ovid Medline Daily Update, Ovid Medline in process & other non-indexed citations, Ovid Embase (1980 to present), The Cochrane Library (most current issue) and Web of Science. No language restrictions will be applied.

                          In addition, the bibliographies of any eligible articles identified will be checked for additional references and citation searches will be carried out for all included references using ISI Web of Knowledge.

                          A search strategy for the electronic literature search in Ovid Medline has already been designed with the support of a librarian/information specialist. This strategy was translated as appropriate for the other databases (for the full search strategies see Appendix A). In addition, we will contact various experts in the field for further eligible studies.

                          Data collection and analyses

                          Selection of methodological research projects

                          A methodological research project will be considered eligible for inclusion in this systematic review if it reviews a cohort of meta-analyses (that is, more than one meta-analyses) that:
                          • ▪ compare pooled effect estimates of meta-analyses of health care interventions according to publication status (that is, published versus unpublished studies or gray literature) or

                          • ▪ examine whether the inclusion of unpublished studies or gray literature impacts the overall findings or conclusions of a meta-analyses

                          We will consider ‘published’ articles to be manuscripts that appeared in peer-reviewed journals. Our working definition of gray literature will correspond to the definitions used by the authors of eligible methodological research projects and which also conforms to the definition of ‘gray literature’ described earlier in this protocol (see Background). A meta-analysis is defined as the calculation of a summary estimate of treatment effect by pooling the results of two or more studies.

                          Data extraction

                          A specifically designed data extraction form will be developed and two reviewers will independently extract all relevant data from eligible methodological research projects. The following information will be collected:
                          1. 1.
                            Characteristics of the methodological research project
                            1. a.

                              Baseline data (for example, author names, affiliation, language and year of publication, funding, type of report (for example, full publication, abstract))

                               
                            2. b.

                              Area of health care/medical specialty

                               
                            3. c.

                              Number of meta-analyses included

                               
                             
                          2. 2.
                            Characteristics of the meta-analyses included in the methodological research project
                            1. a.

                              Type of meta-analyses (for example, individual patient data meta-analyses)

                               
                            2. b.

                              Number of studies included in meta-analyses (overall, median, range)

                               
                            3. c.

                              Number of participants included in meta-analyses (overall, median, range)

                               
                            4. d.

                              Main purpose of meta-analyses (efficacy versus safety)

                               
                            5. e.

                              Source used to retrieve unpublished studies, studies published in the gray literature and published studies

                               
                             
                          3. 3.
                            Characteristics of the studies included in the meta-analyses
                            1. a.

                              Number of unpublished studies, studies published in the gray literature and published studies

                               
                            2. b.

                              Number of participant in unpublished studies, in studies published in the gray literature and in published studies

                               
                            3. c.

                              Number of statistically significant positive or negative unpublished studies, studies published in the gray literature and published studies

                               
                            4. d.

                              Type of unpublished studies (for example, RCTs, observational studies), studies published in the gray literature (for example, abstracts, dissertation, letter, book chapters) and published studies (for example, RCTs, observational studies)

                               
                            5. e.

                              Year of publication of unpublished studies, studies published in the gray literature and published studies

                               
                            6. f.

                              Language and country of unpublished studies, studies published in the gray literature and published studies

                               
                            7. g.

                              Funding source of unpublished studies, studies published in the gray literature and published studies

                               
                            8. h.

                              Type of data source in which gray, unpublished and published studies were identified

                               
                            9. i.

                              Methodological quality (for example, blinding, follow-up time, sample size calculation) of unpublished studies, studies published in the gray literature and published studies (this aspect can only be evaluated if the methodological research project provides enough information)

                               
                             

                          Assessment of validity

                          We will systematically consider the validity and generalizability of the identified evidence provided by each of the methodological research projects by evaluating the following aspects:
                          1. 1.
                            Internal validity:
                            1. a.

                              Role of confounding factors: The results of published studies may differ from those of unpublished studies because of factors other than publication status, such as study design, type of participants, characteristics of the intervention, and methodological quality; in this context, did the researcher of the meta-analyses select comparison groups that were matched (for example, did the unpublished studies or studies published in the gray literature share similar aims, designs, and sample sizes as the published ones)?; if not, were suitable adjustments for potentially confounding factors made?

                               
                            2. b.

                              Definition of publication status: Are explicit criteria given to categorize or define unpublished studies, studies published in the gray literature and published studies?

                               
                            3. c.

                              Selection process: Are search criteria given to identify unpublished studies, studies published in the gray literature and published studies?

                               
                             
                          2. 2.
                            External validity (generalizability):
                            1. a.

                              Did the researcher of the methodological research project select a broad-ranging sample of meta-analyses that reflect the current literature in the field of interest (for example, in terms of size, diversity of topic)?

                               
                            2. b.

                              How was the sample determined (for example, random sample)?

                               
                            3. c.

                              Did two researchers carry out data extraction independently?

                               
                            4. d.

                              Did the researchers provide a complete dataset (regarding the characteristics of the methodological research project and included meta-analyses)?

                               
                             

                          Outcome measures

                          1. 1.

                            The extent to which the effect estimate in a meta-analyses changes with the inclusion or exclusion of unpublished studies and gray literature (quantitative measurement). If possible, we will calculate a ratio of risks or odds ratios between the results of unpublished studies and studies published in the gray literature and the results of published studies and estimate the percentage change (pooled risk ratio from unpublished studies and gray literature divided by pooled risk ratio from published studies). A weighted pooled overall estimate will be calculated taking into account number of studies, participants and events.

                             
                          2. 2.
                            The impact of the inclusion of unpublished studies or studies published in the gray literature on conclusions of meta-analyses (qualitative measurement). The impact will be estimated by calculating the proportion of meta-analyses which show a change in their conclusions according to publication status of the included studies; categorization will be as follows:
                            1. a.

                              Change from negatively significant to positively significant

                               
                            2. b.

                              Change from inconclusive to positively significant

                               
                            3. c.

                              Change from positively significant to inconclusive

                               
                            4. d.

                              Change from negatively significant to inconclusive

                               
                            5. e.

                              Change from inconclusive to negatively significant

                               
                            6. f.

                              Change from positively significant to negatively significant

                               
                            7. g.

                              Change from not clinical relevant to clinical relevant

                               
                            8. h.

                              Change from not clinical relevant to inconclusive

                               
                            9. i.

                              Change from clinical relevant to inconclusive

                               
                            10. j.

                              Change from clinical relevant to not clinical relevant

                               
                            11. k.

                              Change from inconclusive to clinical relevant

                               
                            12. l.

                              Change from inconclusive to not clinical relevant

                               
                             

                          Significance and clinical relevance will be defined according to the definitions provided in the methodological research project.

                          Unit of analyses issues

                          The anticipated unit of analyses is the meta-analyses included in the methodological research project.

                          Assessment of heterogeneity

                          Heterogeneity for pooled outcome measures will be assessed by standard methods including Chi2-test and calculation of the I2 value [9].

                          Assessment of reporting biases

                          Funnel plots will be used to assess the association between point estimates of log odds ratio (a measure of extent of association between meta-analyses’ characteristics and change in summary estimates) and a measure of precision if more than ten methodological research projects provide necessary information. Funnel plots will be visually assessed and appropriate formal statistical tests following recommendations formulated by Sterne et al. will be used to test for asymmetry [5]. In the instance of suspected reporting bias authors will be contacted.

                          Data synthesis

                          Data synthesis will involve a combination of descriptive and statistical summaries of the impact of the inclusion or exclusion of unpublished studies and gray literature on the results of meta-analyses (identified by methodological research projects).

                          The decision on whether or not to combine the results of the included methodological research projects will depend on the assessment of heterogeneity. Where methodological research projects will be judged to be sufficiently homogenous in their design a meta-analyses of these research projects will be carried out. The estimated ratios of unpublished and published in the gray literature only versus published treatment effects generated from each methodological evaluation will then be used to summarize the overall difference in risk ratios between unpublished and published in the gray literature only and published studies. The 95% confidence interval for the combined effect will be estimated using a random effects model.

                          Subgroup analyses and investigation of heterogeneity

                          The following subgroup analyses are planned:
                          1. 1.
                            On the level of the methodological research project
                            1. a.

                              Number of meta-analyses included in the methodological research project

                               
                            2. b.

                              Number of participants included in the methodological research project

                               
                             
                          2. 2.
                            On the level of the meta-analyses
                            1. a.

                              Number of studies (unpublished studies versus studies published in the gray literature versus both)

                               
                            2. b.

                              Number of participants included in studies (unpublished studies versus studies published in the gray literature versus both)

                               
                            3. c.

                              Design of studies (unpublished studies versus studies published in the gray literature versus both)

                               
                            4. d.

                              Source of database: gray literature (for example, conference abstracts or research letters) published in an easily accessible database versus unpublished studies for which immense efforts are required to be identified (for example, contact with pharmaceutical industry)

                               
                            5. e.

                              Type of research work (drug versus non-drug studies, clinical research versus basic research)

                               
                            6. f.

                              Area of health care

                               
                             

                          Sensitivity analyses

                          No sensitivity analyses are planned. However, should the instance arise where a methodological research project with doubtful eligibility is identified (as determined by individual review and analyses of the validity of the methodological research project), sensitivity analyses may be undertaken. Possible sensitivity analyses may be based on the following:
                          • ▪ Methodological quality/validity of the methodological research project; only methodological research projects with low risk of bias will be considered. A methodological research project will be of high risk of bias when the external validity and/or the validity of the included meta-analyses are doubtful.

                          Discussion

                          This systematic review seeks to comprehensively synthesize the growing body of research that is related to the impact of including unpublished studies and studies published in the gray literature in meta-analyses. By considering multiple characteristics and potential confounders related to unpublished studies and studies published in the gray literature, we hope to identify sufficient evidence to conclude whether (or to what extent) inclusion of unpublished studies and studies published in the gray literature has an impact on the pooled effect estimates and the conclusions from a meta-analyses. The findings, including risk factors for unpublished studies and studies published in the gray literature, will have important implications for researchers conducting meta-analyses since they need to be informed about the impact and extent of (not) including unpublished and gray studies in meta-analyses. In addition, this systematic review in combination with the results of other systematic reviews that are part of the OPEN Project will serve to raise awareness about the impact of publication bias and the complexity of this issue. These reviews will also serve as a foundation for a recommendations workshop which will enable key members of the biomedical research community (for example, funders, research ethics committees, and journal editors) to develop future policies and guidelines to lessen the frequency of non-publication and related biases.

                          We acknowledge that more than half of all systematic reviews do not involve meta-analysis in their analyses. Despite the fact that our main outcomes focus on the impact of unpublished and gray studies on pooled effect estimates in meta-analyses, our findings will also be valuable for systematic reviews. It is obvious, if we find a statistical difference in the pooled effect estimates in meta-analyses, it is also likely that gray and unpublished literature impacts descriptive results of systematic reviews. Beside effect estimates, we will also evaluate differences in the methodological quality and study characteristics (such as number of participants, language or methodological quality) between unpublished, gray and published studies. These results will also be valuable for systematic reviews to appraise the potential impact of publication bias.

                          Appendix A

                          Search Strategy for OvidSP MEDLINE (search strategy will be adapted for other databases)

                          Line 10 to 14 and 16 to 35 are not shown because they are part of the search strategy for our 1st systematic review [8] and, therefore, have no consequence for this search.
                          1. 1.

                            exp Publishing/sn

                             
                          2. 2.

                            *publishing/

                             
                          3. 3.

                            publication bias/

                             
                          4. 4.

                            selection bias/

                             
                          5. 5.

                            exp manuscripts as topic/

                             
                          6. 6.

                            ((data or finding? or information or evidence or study or studies or trial? or paper? or article? or report* or literature or work or manuscript? or abstract* or result?) adj6 (unpublish* or un-publish* or unreport* or un-report* or nonpublish* or non-publish* or nonpublicat* or non-publicat* or (publication? adj3 rate?) or "not publish*")).ti,ab.

                             
                          7. 7.

                            (underreport* or under-report* or selective report* or selective publish* or selective publicat* or (final* adj2 (report* or publish* or publicat* or manuscript? or paper? or article?)) or (full? adj2 (report* or publish* or publicat* or manuscript? or paper? or article?)) or (subsequent* adj2 (report* or article? or paper? or publi* or manuscript?)) or (sub-sequent* adj2 (report? or article? or paper? or publi* or manuscript?)) or (complete* adj2 (report* or article? or paper? or publish* or publicat* or manuscript?))).ti,ab.

                             
                          8. 8.

                            (bias* adj3 (publish* or publicat*)).ti,ab.

                             
                          9. 9.

                            or/1-8

                             
                          10. 15.

                            exp animals/ not humans/

                             
                          11. 36.

                            meta-analysis as topic/

                             
                          12. 37.

                            Guidelines as Topic/ or Practice Guidelines as Topic/

                             
                          13. 38.

                            exp Clinical Trials as Topic/

                             
                          14. 39

                            meta-analysis.pt.

                             
                          15. 40.

                            (guideline or practice guideline).pt.

                             
                          16. 41.

                            (guideline? or metaanaly* or meta-analy* or metanaly* or meta-synthe* or metasynthe* or meta-regressi* or metaregressi*).ti,ab.

                             
                          17. 42.

                            (systematic* adj3 (review* or overview*)).ti,ab.

                             
                          18. 43.

                            exp Technology Assessment, Biomedical/

                             
                          19. 44.

                            (health technology assessment? or HTA).ti,ab.

                             
                          20. 45.

                            or/36-44

                             
                          21. 46.

                            9 and 45

                             
                          22. 47.

                            46 not 15

                             
                          23. 48.

                            ((implication? or impact? or influenc* or effect? or differen*) adj6 (publication bias* or unpublish* or un-publish* or unreport* or un-report* or nonpublish* or non-publish* or nonpublicat* or non-publicat* or "not publish*")).ti,ab.

                             
                          24. 49.

                            ((implication? or impact? or influenc* or effect? or differen*) adj6 (selective report* or selective publish* or selective publicat* or (final* adj2 (report* or publish* or publicat* or manuscript? or paper? or article?)) or (full? adj2 (report* or publish* or publicat* or manuscript? or paper? or article?)) or (subsequent* adj2 (report* or article? or paper? or publi* or manuscript?)) or (sub-sequent* adj2 (report? or article? or paper? or publi* or manuscript?)) or (complete* adj2 (report* or article? or paper? or publish* or publicat* or manuscript?)))).ti,ab.

                             
                          25. 50.

                            ((unpublish* or un-publish* or unreport* or un-report* or nonpublish* or non-publish* or nonpublicat* or non-publicat* or "not publish*") adj6 publish*).ti,ab.

                             
                          26. 51.

                            (underreport* or under-report* or selective report* or selective publish* or selective publicat* or (final* and (report* or publish* or publicat* or manuscript? or paper? or article?)) or (full? and (report* or publish* or publicat* or manuscript? or paper? or article?)) or (subsequent* and (report* or article? or paper? or publi* or manuscript?)) or (sub-sequent* and (report? or article? or paper? or publi* or manuscript?)) or (complete* and (report* or article? or paper? or publish* or publicat* or manuscript?))).ti.

                             
                          27. 52.

                            (unpublish* or un-publish* or unreport* or un-report* or nonpublish* or non-publish* or nonpublicat* or non-publicat* or "not publish*" or bias*).ti.

                             
                          28. 53.

                            or/48-52

                             
                          29. 54.

                            47 and 53

                             
                          30. 55.

                            47 and (3 or 4)

                             
                          31. 56.

                            47 and (36 or 37)

                             
                          32. 57.

                            56 and (6 or 7)

                             
                          33. 58.

                            *meta-analysis as topic/

                             
                          34. 59.

                            *Guidelines as Topic/ or *Practice Guidelines as Topic/

                             
                          35. 60.

                            47 and (58 or 59)

                             
                          36. 61.

                            (6 or 7) and (3 or 8)

                             
                          37. 62.

                            54 or 55 or 57 or 60 or 61

                             
                          38. 63.

                            (unpublish* or un-publish* or unreport* or un-report* or nonpublish* or non-publish* or nonpublicat* or non-publicat* or "not publish*").ti,ab.

                             
                          39. 64.

                            7 or 63

                             
                          40. 65.

                            62 and 64

                             
                          41. 66.

                            36 or 37

                             
                          42. 67.

                            3 and 66

                             
                          43. 68.

                            65 or 67

                             
                          44. 69.

                            68 not 15

                             
                          45. 70.

                            remove duplicates from 69

                             

                          Appendix B

                          OPEN Consortium

                          Table 1 shows OPEN Consortium.
                          Table 1

                          OPEN Consortium

                          Contributor

                          Participating Institution

                          Antes, Gerd

                          German Cochrane Center, Institute of Medical Biometry and Medical Informatics University Medical Center, Freiburg, Germany

                          Bassler, Dirk

                          Center for Pediatric Clinical Studies, University Medical Center Tuebingen, Germany

                          Bertele, Vittorio

                          Department of Epidemiology, Mario Negri Institute for Pharmacological Research, Italy

                          Bonfill, Xavier

                          The Clinical Epidemiology & Public Health Department at the Hospital de la Santa Creu i Sant Pau, Spain

                          Bouesseau, Marie-Charlotte

                          World Health Organization, Geneva, Switzerland

                          Boutron, Isabelle

                          INSERM U738 research unit, Paris Descartes University, Paris, France

                          Gallus, Silvano

                          Department of Epidemiology, Mario Negri Institute for Pharmacological Research, Italy

                          Garattini, Silvio

                          Department of Epidemiology, Mario Negri Institute for Pharmacological Research, Italy

                          Ghassan, Karam

                          World Health Organization, Geneva, Switzerland

                          La Vecchia, Carlo

                          Department of Epidemiology, Mario Negri Institute for Pharmacological Research, Italy

                          Lang, Britta

                          German Cochrane Center, Institute of Medical Biometry and Medical Informatics University Medical Center, Freiburg, Germany

                          Littmann, Jasper

                          CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical Scholl, Hannover, Germany

                          Kleijnen, Jos

                          Kleijnen Systematic Reviews Ltd., York, United Kingdom

                          Kulig, Michael

                          Federal Joint Committee, Berlin, Germany

                          Malicki, Mario

                          University of Split School of Medicine, Split, Croatia

                          Marusic, Ana

                          University of Split School of Medicine, Split, Croatia

                          Meerpohl, Joerg

                          German Cochrane Center, Institute of Medical Biometry and Medical Informatics University Medical Center, Freiburg, Germany

                          Mueller, Katharina Felicitas

                          Center for Pediatric Clinical Studies, University Medical Center Tuebingen, Germany

                          Pardo, Hector

                          The Clinical Epidemiology & Public Health Department at the Hospital de la Santa Creu i Sant Pau, Spain

                          Perleth, Matthias

                          Federal Joint Committee, Berlin, Germany

                          Ravaud, Philippe

                          INSERM U738 research unit, Paris Descartes University, Paris, France

                          Reis, Andreas

                          World Health Organization, Geneva, Switzerland

                          Schmucker, Christine

                          German Cochrane Center, Institute of Medical Biometry and Medical Informatics University Medical Center, Freiburg, Germany

                          Schwarzer, Guido

                          German Cochrane Center, Institute of Medical Biometry and Medical Informatics University Medical Center, Freiburg, Germany

                          Strech, Daniel

                          CELLS (Centre for Ethics and Law in Life Sciences), Hannover Medical Scholl, Hannover, Germany

                          Trinquart, Ludovic

                          INSERM U738 research unit, Paris Descartes University, Paris, France

                          Urrútia, Gerard

                          The Clinical Epidemiology & Public Health Department at the Hospital de la Santa Creu i Sant Pau, Spain

                          Von Elm, Erik

                          German Cochrane Center, Institute of Medical Biometry and Medical Informatics University Medical Center, Freiburg, Germany

                          Wager, Elizabeth

                          Sideview, Princes Risborough, United Kingdom

                          Wieland, Alexandra

                          Federal Joint Committee, Berlin, Germany

                          Wolff, Robert

                          Kleijnen Systematic Reviews Ltd., York, United Kingdom

                          Declarations

                          Acknowledgements

                          We thank Patrick Oeller and Laura Cabrera for their input during development and piloting of our data extraction form.

                          The OPEN Project (http://​www.​open-project.​eu) is funded by the European Union Seventh Framework Programme (FP7 - HEALTH.2011.4.1-2) under grant agreement n° 285453.

                          Authors’ Affiliations

                          (1)
                          German Cochrane Center, Institute of Medical Biometry and Medical Informatics, University Medical Center Freiburg
                          (2)
                          Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel
                          (3)
                          Department of Clinical Epidemiology and Biostatistics, McMaster University
                          (4)
                          Institute of Medical Biometry and Medical Informatics, University Medical Centre Freiburg
                          (5)
                          Centre for Paediatric Clinical Studies, University Medical Center Tuebingen
                          (6)
                          Cochrane Switzerland, IUMSP, University Hospital Lausanne

                          References

                          1. Center f or Reviews and Dissemination: Undertaking systematic reviews of research on effectiveness: CRD’s guidance for those carrying out or commissioning reviews. York: University of York; 2001.
                          2. Higgins JPT, Green S: Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011. http://​www.​cochrane-handbook.​org
                          3. Song F, Eastwood A, Gilbody S, Duley L, Sutton AJ: Publication and related bias. Health Technol Assess 2000, 4:1e115.
                          4. McAuley L, Pham B, Tugwell P, Moher D: Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analysis. Lancet 2000, 356:1228–1231.PubMedView Article
                          5. Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, Carpenter J, Rucker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JP: Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011, 343:d4002.PubMedView Article
                          6. Hopewell S, McDonald S, Clarke M, Egger M: Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev 2007, 2:MR000010.PubMed
                          7. Cook D, Guyatt GH, Ryan G: Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA 1993, 269:2749–2753.PubMedView Article
                          8. Portalupi S, von Elm E, Schmucker C, Lang B, Motschall E, Schwarzer G, Gross IT, Scherer RW, Bassler D, Meerpohl JJ: Protocol for a systematic review on the extent of non-publication of research studies and associated study characteristics. Systematic Reviews 2013, 2:2.PubMedView Article
                          9. Higgins JP, Thompson SG, Deeks JJ, Altman DG: Measuring inconsistency in meta-analyses. BMJ 2003, 327:557–560.PubMedView Article

                          Copyright

                          © Schmucker et al.; licensee BioMed Central Ltd. 2013

                          This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://​creativecommons.​org/​licenses/​by/​2.​0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.