Skip to main content

Dealing with predatory journal articles captured in systematic reviews



Systematic reviews appraise and synthesize the results from a body of literature. In healthcare, systematic reviews are also used to develop clinical practice guidelines. An increasingly common concern among systematic reviews is that they may unknowingly capture studies published in “predatory” journals and that these studies will be included in summary estimates and impact results, guidelines, and ultimately, clinical care.


There is currently no agreed-upon guidance that exists for how best to manage articles from predatory journals that meet the inclusion criteria for a systematic review. We describe a set of actions that authors of systematic reviews can consider when handling articles published in predatory journals: (1) detail methods for addressing predatory journal articles a priori in a study protocol, (2) determine whether included studies are published in open access journals and if they are listed in the directory of open access journals, and (3) conduct a sensitivity analysis with predatory papers excluded from the synthesis.


Encountering eligible articles published in presumed predatory journals when conducting a review is an increasingly common threat. Developing appropriate methods to account for eligible research published in predatory journals is needed to decrease the potential negative impact of predatory journals on healthcare.

Peer Review reports

Systematic reviews are intended to critically appraise and synthesize the results of entire bodies of literature. In healthcare, they may be used in the development of clinical practice guidelines. The utility of systematic reviews, however, depends on the quality of the included studies [1]. An emerging methods concern among systematic review teams is that they may unknowingly capture studies published in “predatory” journals. This may degrade the quality of a systematic review, and in medicine, has the potential to create downstream effects that negatively impact patient care.

To the unacquainted, predatory journals are “entities that prioritize self-interest at the expense of scholarship and are characterized by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices” [2]. Predatory journals often exploit the open access (OA) publishing model, where copyrights are typically retained by authors, and work is free to access and build upon [3]. OA publishing commonly requires authors to pay an article processing charge (APC) if their paper is accepted for publication [4]. Predatory journals may promote themselves as “OA” to earn a profit from APCs, but without meeting expected publication best practices. For example, they may not provide peer review [5], meaning work published is not vetted. Peer review can improve the quality and usability of research findings. If a predatory journal states it conducts peer review, but it is not occurring, this has the potential to create bias. Examples of the public making medical decisions based on articles published in predatory journals have been noted previously [2, 6]. Further, a study examining 1907 biomedical studies in presumed predatory journals demonstrated that less than half of the included studies reported having ethics approval, yet more than two million humans were included in the predatory publications [7].

Research published in predatory journals is often difficult to retrieve through database searches as these journals are indexed less commonly than traditional journals, suggesting that when legitimate quality research gets published in predatory journals, patient participation in research, use of animals in research, and funds used for the research including paying APCs, may be wasteful. The quality of articles published in predatory journals has been studied in a sample of 1907 biomedical studies in presumed predatory journals [7]. Overall, the quality of articles was far worse than research published in non-predatory journals. Within this sample, however, there were articles that were of similar quality to research published in non-predatory journals. For example, 14% of trials (n = 13) were registered. Being unable to retrieve these articles due to a lack of indexing in common databases, or choosing to disregard these trials solely due to being published in a predatory journal may impact the conclusions drawn during a systematic review and does not fulfill the goal of a systematic review in providing a comprehensive overview of all eligible studies. The counterpoint to this is that availability of findings in predatory journals also poses a threat, since this could result in unvetted research being used, and in the case of systematic reviews, potentially then integrated into clinical decision making. A recent study reported that articles in suspected predatory journals received fewer citations than those in legitimate journals. The authors concluded that the average predatory journal article has very little effect on the research of others and has limited readership [8]. We disagree with their conclusion since (1) failure to retrieve quality work published in a predatory journal contributes to location bias and publications bias of which detrimental effects are well documented [9,10,11] and (2) retrieving poorly conducted research published in presumed predatory outlets that have not been vetted for accuracy, quality, or ethical approval poses harms to patients if applied by clinicians intending to use evidence-informed decision making [2]. Results from articles published in predatory journals have been included in systematic reviews previously and can alter findings and recommendations made based on knowledge synthesis [12, 13]. The presence of predatory journals in databases has been increasing in recent years with Google Scholar and PubMed, one of the world’s leading biomedical databases, both including articles from predatory journals [12, 14].

Two of us (DBR and BS) recently conducted a systematic review and network meta-analysis aimed to compare the effectiveness of psychosocial therapies among people receiving opioid agonist treatment [15]. The search (see for this review was developed by an experienced medical information specialist (BS) in consultation with the review team. The MEDLINE strategy was peer reviewed using the PRESS guideline. The search did not have any language restrictions; however, eligible articles had to be published in English or French. A total of 71 RCTs were eligible for inclusion in the review. When piloting the risk of bias assessments, two reviewers noticed one trial that stood out as being particularly poorly reported [16]. This study had a high risk of bias and included results that were discrepant between the text and tables, as well as grammar and spelling errors.

We reviewed the website of the journal that published the low-quality trial [16] and the journal was presumed to be predatory. Based on our concerns with this trial [16], we decided to review all included trials to determine if journals were subscription-based, hybrid (with subscriptions and the option for open access publishing), or exclusively open access. For any OA journals, we checked to see if the journal was listed in the Directory of Open Access Journals (DOAJ) and noted its presence or absence. Eight of the 71 included trials were published in OA journals; four (50%) of these OA journals were not listed in the DOAJ and had more than 1 year’s worth of research content. These journal websites also contained spelling errors and atypical statements such as the journal being approved by the minister of health. The articles were published in 2013, 2017, 2017, and 2018 and each of the four studies had a high risk of bias. These trials published in potentially predatory journals were retained in our review to capture all relevant evidence and reduce the risk of publication bias in our review. However, a sensitivity analysis that excluded these studies was then conducted and no differences were found as compared to the primary analysis.

We suspect our experience encountering presumed predatory journals during the conduct of a systematic review is increasingly common. This begs the question: what actions should systematic review teams take to deal with presumed predatory journal articles? We know of no applicable formal guidance. In Table 1, we list some suggested actions. To fulfill the goals of a systematic review by critically appraising and synthesizing results from an entire body of literature, the inclusion of articles published in predatory journals is necessary. Systematic reviews should apply standardized approaches to identify articles that are eligible for a systematic review, including those published in predatory journals and conduct thorough risk of bias assessments to determine the impact of these articles in the synthesis of findings. Facilitating sensitivity analyses and considering results both with and without research from predatory journals may be an important step for future systematic reviews. Failure to search for and include articles published in potentially predatory journals contributes to research waste, publication bias, and incomplete evidence synthesis. Suggested approaches are noted during the protocol and review stage for systematically managing and including eligible articles (see Table 1). During the review, the focus of suggested actions center on assessing the likelihood that an article is published in a predatory journal. The suggested approach involves checking multiple websites to determine if an article (1) is published in an OA journal, (2) is listed in the DOAJ, and (3) is a member of COPE (Committee On Publication Ethics). These suggested steps can be completed at the time of data extraction to decrease additional workload. These recommendations allow articles in potentially predatory journals to be included in systematic reviews while taking into consideration the poor quality that is common among articles published in potentially predatory journals.

Table 1 Suggestions for how systematic reviewers can deal with predatory journals

Planning to search for, address, and include articles in potentially predatory journals prior to beginning a systematic review is important to minimize bias. The preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) is a checklist intended to facilitate the preparation and reporting of a robust protocol for systematic reviews. PRISMA-P includes items related to search strategy, data collection, and risk of bias [18]. Updates of PRISMA-P may consider incorporating data collection for the type of journals included in systematic reviews to systematically assess the presence of potentially predatory journals in future systematic reviews and meta-analyses. Assuring that appropriate methods are in place to account for eligible research published in predatory journals while conducting systematic reviews is necessary to decrease the potential negative impact of predatory journals on healthcare. Future research that considers the best approach to manage the inclusion of articles published in potentially predatory journals will allow for increased consistency across systematic reviews.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.



Directory of Open Access Journals


Open access


Preferred reporting items for systematic review and meta-analysis protocols


Article processing charge


Peer review of electronic search strategies


  1. Higgins JP, Thomas J, Chandler J. In: Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane handbook for systematic reviews of interventions. Chichester: Wiley; 2019.

  2. Grudniewicz A, Moher D, Cobey KD, Bryson GL, Cukier S, Allen K, et al. Predatory journals: no definition, no defence. Nature. 2019;576(7786):210–2.

    Article  CAS  PubMed  Google Scholar 

  3. Suber P. A very brief introduction to open access. Advancing your research skills for the digital age. 2010 Paper 7.

  4. Brown PO CD, Chakravarti A, Cohen B, Delamothe T, Elsen M, Grivell L et al.,. Bethesda Statement on open access publishing. Harvad Library Office for Scholarly Communication Web site. Published 2003. Accessed 1 Dec 2019.

  5. Bohannon J. Who's afraid of peer review? Science. 2013;342(6154);60–5.

  6. Lalu M. Predatory journals prey on public and patients. Web site. Published 2017. Accessed 26 May 2021.

  7. Moher D, Shamseer L, Cobey KD, Lalu MM, Galipeau J, Avey MT, et al. Stop this waste of people, animals and money. Nature News. 2017;549(7670):23–5.

    Article  CAS  Google Scholar 

  8. Björk BC, Kanto-Karvonen S, Harviainen JT. How frequently are articles in predatory open access journals cited. Publications. 2020;8(2):17.

    Article  Google Scholar 

  9. Sampson M, Barrowman NJ, Moher D, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56:943–55.

    Article  Google Scholar 

  10. Boutron I, Page MJ, Higgins JP, et al. Chapter 7: Considering bias and conflicts of interest among the included studies. In: JPT H, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Chichester: Cochrane; 2021. Available from

  11. Murad MH, Chu H, Lin L, Wang Z. The effect of publication bias magnitude and direction on the certainty in evidence. BMJ Evid Based Med. 2018;23(3):84–6.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Ross-White A, Godfrey CM, Sears KA, Wilson R. Predatory publications in evidence syntheses. J Med Libr Assoc. 2019;107(1):57–61.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Hayden JA, Ellis J, Ogilvie R, Boulos L, Stanojevic S. Meta-epidemiological study of publication integrity, and quality of conduct and reporting of randomized trials included in a systematic review of low back pain. J Clin Epidemiol. 2021;134:65–78.

    Article  CAS  PubMed  Google Scholar 

  14. Manca A, Moher D, Cugusi L, Dvir Z, Deriu F. How predatory journals leak into PubMed. CMAJ. 2018 Sep 4;190(35):E1042–5.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Rice DB, Hutton B, Poulin P, Sproule BA, Wolfe D, Moher D, et al. Evaluating comparative effectiveness of psychosocial interventions for persons receiving opioid agonist therapy for opioid use disorder: protocol for a systematic review. BMJ open. 2018 Oct 1;8(10):e023902.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Pashaei T, Shojaeizadeh D, Foroushani AR, Ghazitabatabae M, Moeeni M, Rajati F, et al. Effectiveness of relapse prevention cognitive-behavioral model in opioid-dependent patients participating in the methadone maintenance treatment in Iran. Iran J Public Health. 2013;42(8):896–902.

    PubMed  PubMed Central  Google Scholar 

  17. Cukier S, Helal L, Rice DB, Pupkaite J, Ahmadzai N, Wilson M, et al. Checklists to detect potential predatory biomedical journals: a systematic review. BMC Med. 2020;18:1–20.

    Article  Google Scholar 

  18. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;2:349.

    Google Scholar 

Download references


DBR is supported by a Canadian Institutes of Health Research Vanier Graduate Scholarship.

Author information

Authors and Affiliations



DBR and KDC conceptualized the work. All authors contributed to drafting the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Danielle B. Rice.

Ethics declarations

Ethics approval and consent to participate


Consent for publication


Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rice, D.B., Skidmore, B. & Cobey, K.D. Dealing with predatory journal articles captured in systematic reviews. Syst Rev 10, 175 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: