Skip to main content

A comparison between two recommendations to conduct and report systematic reviews on drug’s safety

Abstract

Background

Several recommendations are available to conduct and report a systematic review of adverse drug reactions. This study is aimed at identifying and comparing the methodologies of the two most commonly used recommendations to conduct and report systematic reviews on drug’s safety.

Methods

Two systematic reviews were conducted following the recommendations “Cochrane Handbook for Systematic Reviews of Interventions” and “Systematic Reviews’ Centre for Reviews and Dissemination guidance for undertaking reviews in healthcare.” The methods of each recommendation were characterized, and the results and the discussion of each systematic review were also evaluated.

Results

The methodologies of both recommendations are similar. The review question was structured. Both recommendations suggest to include pre- and post-marketing data. The recommended data sources differed and, consequently, the results of the systematic reviews (37 vs. 35 studies). Other aspects of search literature were identical. Different tools are suggested to evaluate the methodological quality of the included studies. For case reports, both recommendations only report some questions that may be helpful to assess risk of bias. The reporting of the results and discussion is also identical for both recommendations.

Conclusions

Few methodological differences were observed between the analyzed recommendations to conduct a systematic review on drug’s safety. Combining their methods into a single and recognized recommendation could be of great value.

Peer Review reports

Background

A systematic review can constitute an important tool in pharmacovigilance [1]. A rigorous methodology is used to systematically summarize the available evidence on drug’s safety [2]. Data on the common and expected adverse drug reactions can be obtained from clinical trials [3]. Observational studies, case reports, and spontaneous reports are valuable to detect rare and/or long-term adverse drug reactions [4]. Combining this information using a systematic methodology, which identifies, selects, and critically appraises all available evidence, can expand and strengthen drug’s safety profile [1]. Healthcare professionals, patients, and also regulatory authorities can keep up to date and make informed decisions [5].

To conduct and/or report a systematic review, a defined methodology should be a priori selected. The choice of the recommendation to conduct and/or to report a systematic review is dependent on the review question. For instance, if the aim of a systematic review is to study drug’s efficacy or safety, health economics, and diagnostic test accuracy, among others, the methods will be specific and applied to each area [6,7,8]. There are different recommendations to conduct and/or report a systematic review [9]. Nevertheless, each recommendation presents different methodology [9]. In a previous study, between 88 and 99% of the recommendations offer guidance on methods (from eligibility criteria to data analysis) [9]. However, some aspects, such as elaboration of an a priori protocol, definition of background, interpretation and discussion of the results, and the need and time for updating the systematic review, presented several discrepancies [9].

Three recommendations are available to conduct and report a systematic review of drug’s safety [6,7,8]. Each one was developed by an organization created to study and develop guidance on the best synthesis of different types of information [6,7,8]. The “Cochrane Handbook for Systematic Reviews of Interventions,” developed by the Cochrane Collaboration and last updated on March 2011 [6], and the “Systematic Reviews’ Centre for Reviews and Dissemination guidance for undertaking reviews in healthcare,” developed by the Centre for Reviews and Dissemination (CRD) on 2009 [7], were the recommendations most used to conduct and to report systematic reviews in this field [9]. In 2016, the Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA) group also developed a guideline to reporting systematic reviews of adverse drug reactions [5].

This study is aimed at identifying and comparing the methodology of the two most commonly used recommendations to conduct and report a systematic review of adverse drug reactions.

Methods

In a previous work, we had identified three recommendations (“Cochrane Handbook for Systematic Reviews of Interventions,” “Systematic Reviews’ CRD guidance for undertaking reviews in healthcare,” and “Methods Guide for Effectiveness and Comparative Effectiveness Reviews” developed by the Agency for Healthcare Research and Quality (AHRQ)) used to conduct and/or report a systematic review of adverse drug reactions [9]. The recommendations developed by the Cochrane Collaboration [6] and the CRD [7] are the most commonly used and were included to study their methodology. We did not include the recommendation developed by the AHRQ. Although it can be applied to conduct systematic reviews on drug’s safety, this guidance is specific for comparative effectiveness reviews of interventions under the Effective Health Care Program [8].

The two chosen recommendations were named as “A” for “Cochrane Handbook for Systematic Reviews of Interventions” [6] and “B” for “Systematic Reviews’ CRD guidance for undertaking reviews in healthcare” [7]. We conducted two systematic reviews addressing the methodologies defined by the recommendations A and B. Two authors performed both systematic reviews independently, while a third author validated all methodology and results.

At first, a search strategy was defined between consensus among the three authors and replied in each database. The selection of studies and extraction of results were conducted by two authors independently. Disagreements were resolved by discussion and consensus with the third author. The results in each systematic review were analyzed using descriptive analysis and, when it is possible, using meta-analysis. Details of the elaboration of each systematic review are available in the Additional file 1. At the end, the methods of each recommendation were characterized and evaluated. We categorized their methods into “Introduction,” which included “Background,” “Eligibility criteria,” and “Review question”; “Identifying evidence,” including “Type of studies,” “Databases,” “Search strategy,” “Data selection,” “Data extraction,” “Quality assessment,” and “Data synthesis”; and “Reporting,” describing the “Flowchart,” “Characteristics of studies,” “Outcome analysis,” “Quality assessment,” “Discussion,” “Conclusion,” “Funding,” and “Appendix.” Afterwards, a qualitative (exploratory) comparison between the results and impact of these results was performed.

In order to study the influence of different recommendations to conduct a systematic review, the same research question was studied in both systematic reviews. A case study was used to compare both methodologies. We assessed the development of an ophthalmic adverse drug reaction after a suspected medicine exposure. Three regulatory agencies issued a safety alert on the association of non-arteritic anterior ischemic optic neuropathy (NAION) with phosphodiesterase type 5 (PDE5) inhibitors [10]. In order to study such association, we defined a research hypothesis, structured according to PICO strategy: to assess the risk of developing NAION in individuals taking PDE5 inhibitors. The results of both systematic reviews can be found in Additional file 1. Herein, the results of the systematic reviews were compiled into a table.

Results

The methodology suggested by the recommendations A and B to conduct and report a systematic review of adverse drug reactions is summarized in Table 1. We highlighted the sections of the systematic review which differed between both recommendations.

Table 1 Summary of methodology used in each systematic review

In general, the methodology of both recommendations is similar. Both systematic reviews provide a detailed rationale for conducting the review, focusing on the description of the intervention and condition, and their possible association. The review question was structured and follows the PICO (Population, Intervention, Comparator, and Outcome) strategy. The two recommendations suggest to search studies including pre- and post-marketing data on drug’s safety. The bibliographic databases and other sources to search evidence suggested in each recommendation differed. Other aspects of search literature, such as the search strategy, data selection, data extraction, and data synthesis, were identical. To evaluate the methodological quality of the included studies, the Cochrane Collaboration developed three scales which evaluate randomized controlled trials, cohort, and case-control studies. The CRD’s guideline (B) recommends some tools to evaluate the several types of studies. For case reports, both guidelines report some questions that may be helpful to evaluate the risk of bias/quality of the reports. The reporting of the results and discussion is also identical for both recommendations.

Discussion

There are three recommendations to conduct systematic reviews of adverse drug reactions [9]. In order to assess the methodological differences between those recommendations, in this study, two systematic reviews were conducted following the two most commonly used recommendations. Similar approaches in the elaboration of the two systematic reviews of adverse drug reactions were observed.

One of the characteristics that distinguish systematic reviews from narrative reviews is a structured review question [7, 8]. In both systematic reviews, a structured question was presented. It predisposes to a focused hypothesis and can also be useful to define eligibility, search, and inclusion criteria, and presentation of results [11]. Similar to efficacy systematic reviews, systematic reviews focusing on drug’s safety could present a broad or narrow review question [11]. A systematic review studying the association of a class of medicines with a specific adverse drug reaction (for example, the development of retinal detachment after fluoroquinolones exposure [12]) has a narrow focus, while a systematic review studying all adverse drug reactions associated with a class of medicines (for example, the safety profile of ophthalmic anti-angiogenesis inhibitors [13]) has a broad focus [11].

Both recommendations suggested the selection of several types of evidence. These included experimental and observational data. Depending on the review question, each type of sources of information could be the most appropriate. Clinical trials are a robust source of information [3]. Data on common and anticipated adverse drug reactions could be obtained from these studies [3]. Observational studies have some flaws subject of bias in their methodology, such as in the demographic characteristics of the included populations, the follow-up time durations, or the effect size measures used [4]; however, they can provide relevant information on the common, rare, and long-term adverse drug reactions [4]. Examples of observational data include clinical studies in clinical practice, health-administrative database studies, cases and series of case reports, and spontaneous reports [4]. An example of the importance of observational data on drug’s safety assessment is visible in the regulatory decisions. Some safety alerts were issued based only on spontaneous reports [10, 14]. In general, systematic reviews of drug’s efficacy contain data from the highest level of evidence, namely randomized and controlled clinical trials [15]; however, in systematic reviews focusing on drug’s safety, several types of evidence should be included, in order to obtain a more complete and robust drug safety profile. The combination of several types of evidence in safety is already used to make better informed decisions [14]. In 2012, the US Food and Drug Administration issued a safety alert on statins and cognitive side effects. The evidence supporting the regulatory evaluation consisted on randomized controlled trials, observational studies (cohort studies, case-control studies, cross-sectional studies), case reports, and a review of post-marketing spontaneous report database. Thereafter, the adverse drug reaction section of the medicines was updated [14].

A difference which impacts the results of these two systematic reviews was the selection of the bibliographic databases and other sources of information. The authors of the present study only searched in the suggested sources of each recommendation. We also experienced some difficulties in accessing some bibliographic databases. Several studies pointed out the importance of including data from a wide range of bibliographic databases, not only because of their limitations (some databases are more accurate and include newest studies than others), but also because some studies may only be available in one bibliographic database (for instance gray literature is not easily available) [16, 17]. At the present work, we observed some discrepancies between the two systematic reviews in terms of the number of included studies. For instance, two case reports were not included in one of the performed systematic reviews. Nonetheless, we studied the risk of development of a rare adverse drug reaction. Therefore, the volume of available information could be reduced compared with that of a common adverse drug reaction. If the information not found was of higher methodological quality than case reports (such as case-control or cohort studies), the impact of not including this data would be substantial. For instance, in both systematic reviews, the same four observational studies were found. A meta-analysis was conducted based on their results. If a study was missed, the results of this meta-analysis would be different, and consequently, the risk estimate could vary for opposite meanings (risk vs. no risk). Conducting a search using all available sources of information is important, as this will ensure that all relevant data are obtained and evaluated [11]. A systematic review performed by Baudard et al. evaluated the impact of searching on clinical trials registries for additional studies [18]. By analyzing a predefined sample of systematic reviews, this research group found that 52% did not report a search on clinical trials registries [18]. After performing searches on clinical trial registries, Baudard et al. found 122 additional randomized controlled trials [18]. If these studies had been included in meta-analysis, the weight of studies would be changed to almost 60% in some systematic reviews [18]. Other study conducted by Franco et al. found that 73% of reviews have issues in the definition of literature searches [19].

The methodological quality assessment for experimental and observational studies, such as cohort and case-control studies, is possible due to a variety of available tools. However, few or no tools are available to assess other types of observational studies, case reports, health-administrative database studies, case and series of case reports, and spontaneous reports [20]. Since a hierarchy of evidence is not yet defined to assess drug’s safety, several studies questioned the usefulness of these tools once a combination of several types of studies is included in a systematic review and an evaluation of all is not possible [21]. In addition, the majority of the available risk of bias tools are not prepared to assess studies on adverse drug reactions [22]. A systematic review evaluating the quality of reporting in systematic reviews of drug’s safety studies found that a large proportion of the analyzed systematic reviews failed in reporting risk of bias assessment [23]. Nevertheless, some efforts are being made to improve the methodological quality assessment of studies reporting drug’s safety. In addition, a recent study has described a new risk of bias tool to use when conducting systematic reviews of randomized controlled trials, cohort studies, case-control studies, and nested case-control studies describing adverse drug reactions [22].

Some studies assessed the quality of systematic reviews reporting adverse drug reactions. In general, systematic reviews reporting adverse drug reactions failed methodologically [23,24,25]. Definition of adverse drug reaction, design of literature search, bibliographic database choice, and assessment of methodological quality of the included studies are the main divergent steps [23,24,25]. In most of the studies, only a small proportion has good reporting [23,24,25]. In a previous work, we analyzed the methodology used in systematic reviews reporting ophthalmic adverse drug reactions and found the same methodological issues [26]. In 2016, the PRISMA group developed guidance to help reporting systematic reviews of adverse drug reactions, the PRISMA Harms [11]. A study performed by Li et al. evaluated the methods of a sample of systematic reviews, 1 year after the publication of PRISMA Harms [27]. They concluded that a large number of systematic reviews still presented methodological differences [27]. The reinforcement of the use of recommendations to conduct and/or report a systematic review of adverse drug reactions still continues of major importance.

Limitations of this study

This study has some limitations. Only two recommendations were used to perform a comparison on methods of reviewing drug’s safety data. The two systematic reviews were conducted by the authors of the present study. Only an exploratory analysis was performed to compare both recommendations. A statistical analysis of the methodologies of both recommendations will be necessary to better understand the differences among them. In the two systematic reviews, the authors only searched on the recommended bibliographic databases, despite the recommendations suggested that other databases could be included. Therefore, two case reports were not included in one systematic review. This resulted in an example of the non-inclusion of all available evidence and, consequently, differences in the results of the systematic reviews.

Further investigation

Further consideration should be taken on the access of the data, including public bibliographic databases and the selection of all available databases, resulting in a more robust and complete data and, therefore, improving the knowledge provided by the systematic review.

In addition, new tools able to evaluate methodological quality of some studies, such as some type of observational studies and case reports, should be elaborated. Moreover, several sources of information should be recorded and more investigation should be performed to clarify the role of the methodological quality assessment in the context of evaluating the evidence of safety.

Finally, the process of conducting and reporting a systematic review of adverse drug reactions, including the design of the review, data search, selection, extraction, and synthesis, should be transparent and independent.

Conclusions

Few methodological differences were observed among the available recommendations to conduct a systematic review of adverse drug reactions. Combining their methods into a single and recognized recommendation could be of great value. A unique, objective, and easy to apply methodology could improve systematic review’s role in drug safety. Further research should be considered, namely in granting access to the information and in the methodological quality assessment of the included evidence.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Abbreviations

AHRQ:

Agency for Healthcare Research and Quality

CRD:

Centre for Reviews and Dissemination

NAION:

Non-arteritic ischemic optic neuritis

PDE5:

Phosphodiesterase type 5

PICO:

Population, Intervention, Comparator, Outcome

PRISMA:

Preferred Reporting Items for Systematic Reviews and Meta-Analysis

References

  1. EUNETHA. Endpoints used in relative effectiveness assessment – safety; 2015. p. 1–42. http://www.eunethta.eu/sites/default/files/sites/5026.fedimbo.belgium.be/files/WP7-SG3-GL-safety_amend2015.pdf. Accessed 19 Nov 2018

    Google Scholar 

  2. Oxman A, Guyatt G. The science of reviewing research. Ann N Y Acad Sci. 1993;703:125–33.

    Article  CAS  Google Scholar 

  3. Singh S, Loke YK. Drug safety assessment in clinical trials: methodological challenges and opportunities. Trials. 2012;13:138.

    Article  Google Scholar 

  4. Strom BL. Study designs available for pharmacoepidemiology studies. In: Strom BL, Kimmel SE, editors. Textbook of Pharmacoepidemiology. West Sussex: Wiley; 2006. p. 13–24.

    Google Scholar 

  5. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–9.

    Article  Google Scholar 

  6. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0: The Cochrane Collaboration; 2011. Updated March 2011. Available at: www.handbook.cochrane.org

  7. Centre for Reviews and Dissemination. Systematic reviews. CRD’s guidance for undertaking reviews in healthcare. 2009. ISBN: 1900640473.

    Google Scholar 

  8. Agency for Healthcare Research and Quality. Methods guide for effectiveness and comparative effectiveness reviews. Rockville: AHRQ Publication No. 10(14)-EHC063-EF; 2014. https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-methods-guide_overview.pdf. Accessed 19 Nov 2018

    Google Scholar 

  9. Penedones A, Alves C, Batel MF. Recommendations to conduct and report systematic reviews in medical literature: systematic review: Sociedade Portuguesa de Farmacologia Meeting; 2018. http://www.spfarmacologia.pt/LivroAbstracts.pdf. Accessed 19 Nov 2018

  10. Penedones A, Mendes D, Alves C, et al. Drug-induced ocular adverse reactions: review of the safety alerts issued during the last decade. J Ocul Pharmacol Ther. 2015;31(5):258–68.

    Article  CAS  Google Scholar 

  11. Zorzela L, Loke YK, Ioannidis JP, et al. PRISMA harms checklist: improving harms reporting in systematic reviews. BMJ. 2016;352:i157.

    Article  Google Scholar 

  12. Alves C, Penedones A, Mendes D, et al. A systematic review and meta-analysis of the association between systemic fluoroquinolones and retinal detachment. Acta Ophthalmol. 2016;94(5):e251–9.

    Article  CAS  Google Scholar 

  13. Penedones A, Mendes D, Alves C, et al. Safety monitoring of ophthalmic biologics: a systematic review of pre- and postmarketing safety data. J Ocul Pharmacol Ther. 2014;30(9):729–51.

    Article  CAS  Google Scholar 

  14. Alves C, Macedo AF, Batel MF. Sources of information used by regulatory agencies on the generation of drug safety alerts. Eur J Clin Pharmacol. 2013;69(12):2083–94.

    Article  Google Scholar 

  15. The Centre for Evidence-Based Medicine. Oxford centre for evidence-based medicine – levels of evidence. (2009). https://www.cebm.net/2009/06/oxford-centre-evidence-based-medicine-levels-evidence-march-2009/. Accessed 14 Jan 2019.

    Google Scholar 

  16. Falagas ME, Pitsouni EI, Malietzis GA, et al. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2008;22(2):338–42.

    Article  CAS  Google Scholar 

  17. Haddaway NR, Collins AM, Coughlin D, et al. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS One. 2015;10(9):e0138237.

    Article  Google Scholar 

  18. Baudard M, Yavchitz A, Ravaud P, et al. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ. 2017;356:j448.

    Article  Google Scholar 

  19. Franco JVA, Garrote VL, Escobar Liquitay CM, et al. Identification of problems in search strategies in Cochrane Reviews. Res Synth Methods. 2018;9(3):408–16.

    PubMed  Google Scholar 

  20. Glasziou P, Vandenbroucke JP, Chalmers I. Assessing the quality of research. BMJ. 2004;328:39–41.

    Article  Google Scholar 

  21. McIntosh HM, Woolacott NF, Bagnall AM. Assessing harmful effects in systematic reviews. BMC Med Res Methodol. 2004;4:19.

    Article  Google Scholar 

  22. Faillie JL, Ferrer P, Gouverneur A, et al. A new risk of bias checklist applicable to randomized trials, observational studies, and systematic reviews was developed and validated to be used for systematic reviews focusing on drug adverse events. J Clin Epidemiol. 2017;86:168–75.

    Article  Google Scholar 

  23. Zorzela L, Golder S, Liu Y, et al. Quality of reporting in systematic reviews of adverse events: systematic review. BMJ. 2014;348:f7668.

    Article  Google Scholar 

  24. Cornelius VR, Perrio MJ, Shakir SA, et al. Systematic reviews of adverse effects of drug interventions: a survey of their conduct and reporting quality. Pharmacoepidemiol Drug Saf. 2009;18(12):1223–31.

    Article  CAS  Google Scholar 

  25. Mahady SE, Schlub T, Bero L, et al. Side effects are incompletely reported among systematic reviews in gastroenterology. J Clin Epidemiol. 2015;68(2):144–53.

    Article  Google Scholar 

  26. Penedones A, Batel MF. Methodologic assessment of the systematic reviews of ophthalmic adverse drug reactions published in ophthalmology journals: a systematic review. Ophthalmic Res. 2018;60(2):55–68.

    Article  CAS  Google Scholar 

  27. Li L, Xu C, Deng K, et al. The reporting of safety among drug systematic reviews was poor before the implementation of the PRISMA harms checklist. J Clin Epidemiol. 2019;105:125–35.

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

The authors have nothing to declare.

Author information

Authors and Affiliations

Authors

Contributions

AP was involved in the conception and design of the study, analysis and interpretation of the data, and drafting of the paper. CA was involved in the analysis and interpretation of the data. FBM was involved in the conception and design of the study and revising of the paper critically for intellectual content. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ana Penedones.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Results of systematic reviews conducted and reported according to the two most used recommendations on drug’s safety systematic reviews.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Penedones, A., Alves, C. & Batel Marques, F. A comparison between two recommendations to conduct and report systematic reviews on drug’s safety. Syst Rev 8, 238 (2019). https://doi.org/10.1186/s13643-019-1167-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-019-1167-5

Keywords