Defining publication bias: protocol for a systematic review of highly cited articles and proposal for a new framework
Systematic Reviews volume 2, Article number: 34 (2013)
Selective publication of studies, which is commonly called publication bias, is widely recognized. Over the years a new nomenclature for other types of bias related to non-publication or distortion related to the dissemination of research findings has been developed. However, several of these different biases are often still summarized by the term 'publication bias'.
As part of the OPEN Project (To Overcome failure to Publish nEgative fiNdings) we will conduct a systematic review with the following objectives:
- To systematically review highly cited articles that focus on non-publication of studies and to present the various definitions of biases related to the dissemination of research findings contained in the articles identified.
- To develop and discuss a new framework on nomenclature of various aspects of distortion in the dissemination process that leads to public availability of research findings in an international group of experts in the context of the OPEN Project.
We will systematically search Web of Knowledge for highly cited articles that provide a definition of biases related to the dissemination of research findings. A specifically designed data extraction form will be developed and pilot-tested. Working in teams of two, we will independently extract relevant information from each eligible article.
For the development of a new framework we will construct an initial table listing different levels and different hazards en route to making research findings public. An international group of experts will iteratively review the table and reflect on its content until no new insights emerge and consensus has been reached.
Results are expected to be publicly available in mid-2013. This systematic review together with the results of other systematic reviews of the OPEN project will serve as a basis for the development of future policies and guidelines regarding the assessment and prevention of publication bias.
Systematic reviews and meta-analyses of high-quality randomized controlled trials provide a valid summary of the available research findings, and are therefore crucial to evidence-based decision making. It has long been recognized that the identification of the entire relevant research evidence is essential in order to come up with an unbiased and balanced summary. Thus, ideally all research activities conducted should be published and easily identifiable. As a second choice, the published studies should at least represent a random sample of all studies that have been conducted and the decision on publication should not depend on the nature and direction of the results. Only under such circumstances can systematic reviews and meta-analyses live up to their promise of providing unbiased, high-quality evidence for medical decision making. However, it is not always possible to retrieve all eligible evidence for a given topic, as many studies never get published. The phenomenon of non-publication of studies based on the nature and direction of the results has historically been referred to as 'publication bias'[2, 3].
The public perception cannot only be distorted by the non-publication of an entire study, information may also be partially lacking or presented in a way that influences the take-up of the findings, such as selective reporting of outcomes or subgroups or 'data massaging' (for example, the selective exclusion of patients from the analysis). Thus, over recent years a new nomenclature for other types of bias related to the non-publication or distortion in the dissemination process of research findings has been developed, such as 'outcome reporting bias', 'time lag bias', 'location bias'[6, 7], and many more. Nevertheless, all these different aspects are often still referred to as 'publication bias', although up to now no consensus on the definition of publication bias has been reached in the literature. Therefore, in this systematic review, we aim to summarize what is commonly understood by the term 'publication bias' and to propose a new framework on nomenclature of the various aspects of distortion in the dissemination process of research findings.
In terms of the above mentioned controversies regarding the definition of 'publication bias', we will conduct a systematic review with the following objectives:
To systematically review highly cited articles that focus on non-publication of studies and to present the various definitions of biases related to the dissemination of research findings contained in the articles identified.
To develop and discuss a new framework on nomenclature of various aspects of distortion in the dissemination process that leads to public availability of research findings in an international group of experts in the context of the OPEN Project (To Overcome failure to Publish nEgative fiNdings).
This systematic review will be part of the OPEN Project which was developed with the goal of elucidating the scope of non-publication of studies through a series of systematic reviews (for example, see).
Systematic literature search
We will search Web of Knowledge on a given day. We will use the simple search term 'publication bias'. We chose Web of Knowledge because it always follows a ranking according to the total number of citations and therefore allows us to identify the most frequently cited articles. Although we are interested in various aspects of bias in the dissemination process of research findings, our main aim is the identification of different definitions of publication bias and we thus decided that the term 'publication bias' should be part of all publications of interest. No language restrictions will be applied. We will not search any other database or any grey literature since our focus is on highly cited and publicly available articles in order to capture the most used definitions of publication bias.
We will include the 50 most frequently cited articles that focus on biases related to the non-publication or distortion in the dissemination process of research findings from any source and addressed to any audience. We chose the number 50 arbitrarily and will not exclude self-citations because we are interested in the absolute number, independent of the people that cited the work. In order to be included, articles must include the term ‘publication bias’ and provide some form of definition of this phenomenon. We will only include fully published articles.
Two reviewers will independently and in duplicate screen titles and abstracts of search results. If a title or abstract cannot be rejected with certainty by both reviewers, the full text of the paper will be retrieved and assessed for eligibility. Any disagreement among the reviewers will be resolved by discussion and consensus or, if needed, by third-party arbitration.
A specifically designed information extraction form will be developed and pilot-tested. Two reviewers will independently extract all relevant information from each eligible article. The following information will be collected:
o Baseline data (for example, author names, language and year of publication, journal)
o Number of citations in Web of Knowledge and rank
o Definitions of biases related to the dissemination of research findings
o Methods to detect, quantify and adjust for publication bias
o Suggestions to minimize publication bias
o Impact of publication bias
Any disagreement will be resolved by discussion and consensus or, if needed, arbitration by a third reviewer.
Data analysis and reporting
Data synthesis will involve a descriptive summary of the range of definitions given to describe various forms of biases related to the dissemination of research findings. We will report the study according to PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines.
We will propose a draft capturing the ideas and experiences of the core group of authors. This draft will be complemented by our findings from the 50 most frequently cited articles in Web of Knowledge. We will then circulate a draft to an international group of experts that are part of the OPEN consortium. Each expert will review the draft and provide feedback regarding the issues we identified or contribute with other insights. We will continue this process until no additional ideas emerge.
At the end of this process we aim for consensus regarding the range of mechanisms that can distort the dissemination of research findings, its nomenclature and related definitions. We endeavor to provide a framework which focuses on the responsibilities of all players on the various stages of the dissemination process.
The Helsinki Declaration states clearly that 'Authors, editors and publishers all have ethical obligations with regard to the publication of the results of research. Authors have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports. […]. Negative and inconclusive as well as positive results should be published or otherwise made publicly available'. However, many research results never get published. The non-publication of study results is of great importance because it distorts the evidence base for clinical decision-making, which is increasingly based on the synthesis of published research and might therefore lead to patients receiving an ineffective or even harmful treatment[12–15].
This systematic review seeks to give a broad overview of the various definitions currently used to describe biases related to the dissemination of research findings. Based on our findings, together with reflections from an international group of experts (the OPEN consortium), we aim to propose a new framework on nomenclature of various aspects of distortion in the dissemination process of research findings which focuses on the responsibilities of all players. We hope that this player-based framework will help to clarify responsibilities for publication and related bias.
Being part of the OPEN Project this systematic review, together with the results of other systematic reviews of this subject, aims to raise awareness of the significance of bias related to the non-publication or distortion in the publication process of research findings and the complexity of this issue. In addition, these reviews will also provide a foundation for a recommendations workshop, which will enable key members of the biomedical research community (for example, funders, research ethics committees, journal editors, etc.) to develop future policies and guidelines to minimize non-publication and related biases.
To Overcome failure to Publish nEgative fiNdings
Preferred Reporting Items for Systematic reviews and Meta-Analyses.
Higgins JPT, Green S: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011, The Cochrane Collaboration, Available fromhttp://www.cochrane-handbook.org,
Dickersin K: The existence of publication bias and risk factors for its occurrence. JAMA. 1990, 263: 1385-1389. 10.1001/jama.1990.03440100097014.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR: Publication bias in clinical research. Lancet. 1991, 337: 867-872. 10.1016/0140-6736(91)90201-Y.
Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004, 291: 2457-2465. 10.1001/jama.291.20.2457.
Hopewell S, Clarke M, Stewart L, Tierney J: Time to publication for results of clinical trials. Cochrane Database Syst Rev. 2007, MR000011-
Pittler MH, Abbot NC, Harkness EF, Ernst E: Location bias in controlled clinical trials of complementary/alternative therapies. J Clin Epidemiol. 2000, 53: 485-489. 10.1016/S0895-4356(99)00220-6.
Vickers A, Goyal N, Harland R, Rees R: Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998, 19: 159-166. 10.1016/S0197-2456(97)00150-5.
Portalupi S, von Elm E, Schmucker C, Lang B, Motschall E, Schwarzer G, Gross IT, Scherer RW, Bassler D, Meerpohl JJ: Protocol for a systematic review on the extent of non-publication of research studies and associated study characteristics. Systematic reviews. 2013, 2: 2-10.1186/2046-4053-2-2.
Kulkarni AV, Aziz B, Shams I, Busse JW: Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA. 2009, 302: 1092-1096. 10.1001/jama.2009.1307.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009, 151: 264-269. 10.7326/0003-4819-151-4-200908180-00135.
Declaration of Helsinki:Ethical Principles for Medical Research Involving Human Subjects. 2008, World Medical Association, Available from:http://www.wma.net/en/30publications/10policies/b3/,
Egger M, Smith GD: Misleading meta-analysis. BMJ. 1995, 310: 752-754. 10.1136/bmj.310.6982.752.
Chalmers I: Underreporting research is scientific misconduct. JAMA. 1990, 263: 1405-1408. 10.1001/jama.1990.03440100121018.
Cowley AJ, Skene A, Stainer K, Hampton JR: The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction: an example of publication bias. Int J Cardiol. 1993, 40: 161-166. 10.1016/0167-5273(93)90279-P.
Garland EJ: Facing the evidence: antidepressant treatment in children and adolescents. CMAJ. 2004, 170: 489-491.
The OPEN Project (http://www.open-project.eu) is funded by the European Union Seventh Framework Programme (FP7 – HEALTH.2011.4.1-2) under grant agreement n° 285453.
We declare that all authors and contributing members have no competing interests.
DB and JJM conceived of the study. DB and KFM designed the search strategies. KFM drafted the manuscript with the help of DB. KFM, MB, AD, JK, AM, EW, GA, EvE, BL, EM, VG, GS, DA, JJM, and DB critically reviewed the manuscript for important intellectual content. KFM, MB, JK, AM, EW, EvE, DA, JJM and DB will develop the new framework. All authors read and approved the final version before submission. KFM and DB are guarantors.
About this article
Cite this article
Müller, K.F., Briel, M., D’Amario, A. et al. Defining publication bias: protocol for a systematic review of highly cited articles and proposal for a new framework. Syst Rev 2, 34 (2013). https://doi.org/10.1186/2046-4053-2-34