Skip to main content

Signals of adverse drug reactions communicated by pharmacovigilance stakeholders: protocol for a scoping review of the global literature



Signals of adverse drug reactions (ADRs) form the basis of some regulatory risk-minimization actions in pharmacovigilance. Reviews of limited scope have highlighted that such signals are mostly supported by reports of ADRs or multiple types of evidence. The time that elapses between a report of a suspected ADR and the communication of a signal has not been systematically characterized. Neither has the features of reports of suspected ADRs that authors used to support putative causal relationships, although difficulties with establishing causal relationships between medicinal products and adverse events have been highlighted. The objectives of this study will be to describe the evidence underpinning signals in pharmacovigilance, the features of reports of ADRs supporting signals, and the time that it takes to communicate a signal.


We shall retrieve records from PubMed, EMBASE, Web of Science, and PsycINFO (from inception onwards), without language/design restrictions, and apply backward citation screening. We shall hand-search the websites of 35 regulatory agencies/authorities, restricted publications from the Uppsala Monitoring Centre, and drug bulletins. Signals will be requested from the competent stakeholder, if absent from websites. We shall use VigiBase, the World Health Organization’s Global Individual Case Safety Report database, to determine the dates on which ADRs were reported. We shall manage records using EndNote (v. 8.2); one reviewer will screen titles/abstracts and full texts, a second will cross-validate the findings, and a third will arbitrate disagreements. Data will be charted via the Systematic Reviews Data Repository, following the same procedures as for data retrieval. Evidence will be categorized according to the Oxford Centre for Evidence-Based Medicine Levels of Evidence. Features of reports of ADRs will be coded. Tables will display frequencies of types of evidence and features of reports of ADRs. We shall use plots or pictograms (if appropriate) to represent the time from the first report of a suspected ADR to a signal.


We expect the findings from this review will allow a better understanding of global patterns of similarities or differences in terms of supporting evidence and timing of communications and identify relevant research questions for future systematic reviews.

Systematic review registration

Peer Review reports


A systematic review of definitions, later adopted by the Council for International Organizations of Medical Sciences, defined a signal in pharmacovigilance as “information that arises from one or multiple sources (including observations and experiments), which suggests a new potentially causal association, or a new aspect of a known association, between an intervention and an event or set of related events, either adverse or beneficial, which would command regulatory, societal or clinical attention, and is judged to be of sufficient likelihood to justify verificatory and, when necessary, remedial actions” [1, 2].

Based on this definition, a signal can be supported by diverse levels of evidence, ranging from one or more reports of suspected adverse drug reactions (ADRs) to observational studies and randomized clinical trials [3,4,5]. On the other hand, signals of disproportionate reporting (SDRs) [1] can be highlighted solely by statistical associations resulting from ratios of observed versus expected numbers of reports (i.e. disproportionality) [6]. Different stakeholders can detect signals—including regulatory agencies, pharmaceutical companies, academics, independent research organizations, or patients and their carers—and avail themselves of several channels to disseminate them, such as peer-reviewed articles, abstracts, drug bulletins, websites, and restricted publications. Minutes of meetings of committees can also document discussions relating to signals [7].

Comparative assessments of signals discussed at the Pharmacovigilance Risk Assessment Committee have suggested that reports of ADRs often supported signals, but that support from multiple sources of evidence increased the likelihood of regulatory action (e.g. changes to Summaries of Product Characteristics) [8, 9]. Similar investigations carried out in the USA have highlighted the same finding [10, 11]. Regulatory actions, however, appear to be inconsistent across international settings; discrepancies in communications of the same risks of drug-induced harms in healthcare systems have been recorded, and it has been suggested that they may be better understood by characterizing the strength of the underpinning evidence [12, 13].

An important component of detection of signals of ADRs is clinical judgment of the data. The Bradford-Hill viewpoints or guidelines [14] and other structured methods of causality assessment [15] have typically guided clinical judgments in pharmacovigilance, but their intrinsic subjectivity has yielded low interrater agreement across independent assessors [16]. Though some methods have a higher interrater agreement than others, there appears to be no consensus on which ought to be the gold standard [17]. Consequently, regulators’ clinical judgments about features of reports of ADRs have occasionally been criticized by healthcare workers or their professional associations [18,19,20,21]. It has been reported that difficulties in assessing causality result in delays in the withdrawal of medicinal products from the market when fatal adverse events are attributed to their use [22]. Furthermore, the time intervals before signals of ADRs appear and regulatory actions follow are determined by the availability of information about the ADRs at the time of product launch; in fact, these intervals were shorter when ADRs were present in a sample of the Food and Drug Administration’s “drug label” at launch (e.g. severe liver injury with telithromycin or suicidal behaviour with varenicline) and longer when ADRs were absent [23]. Other accounts report that it may take 2–12 years for drug-induced harms to be communicated to healthcare workers after marketing authorization of a medicinal product [24,25,26,27]. These studies have computed the difference between dates of communication of signals and launch dates to measure the time-to-signal, while databases of reports of ADRs, such as VigiBase, could estimate this interval and account for unpublished records.

Over several decades of pharmacovigilance, many definitions of signals have been developed and applied, and different types of studies have been used to detect signals, resulting in a complex body of evidence whose breadth has been only partly characterized. Delays to withdrawals have been investigated systematically, while those relating to the time to signal have been limited in scope. Furthermore, while reports of ADRs remain a staple in signal detection, it is unclear to what extent their features (e.g. time to onset, dose-response relationship, positive de-/re-challenge) have been described or contributed to putative causal relationships. This is relevant when one considers that recent data-driven methods to prioritize signals arising from reports of ADRs are based on the quality of information of the reports themselves and a limited set of features, i.e. the presence of a narrative, the geographical distribution of the reports of ADRs, how recently reports of ADRs have been entered into a database, and a numerical measure of data completeness [28, 29]. Comprehensive overviews of the features of reports of ADRs that support signals may further inform improvements to such methods.

We therefore believe that it is worthwhile to conduct a scoping review, since such reviews are suitable for addressing broad exploratory questions pertaining to types of evidence in an area of research [30,31,32]

The aims of this scoping review are as follows: (a) to provide an evidence synthesis for decision-making at a regulatory level; (b) to collate signals currently in different platforms; (c) to produce an overview of the types and levels of evidence of studies that support signals; (d) to highlight any differences or similarities in assessment of the evidence used to support signals, with a particular focus on reports of ADRs; and (e) to better understand the delays before signals are detected and to clarify whether they have changed over time.


The six-step framework introduced by Arksey and O’Malley [30] and refined by Levac et al. [31] and Colquhoun et al. [32] will set the structure of this protocol.

The review protocol has been registered in the Open Science Framework database (registration number: and is being reported in accordance with the reporting guidance provided in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) statement [33, 34] (see checklist in Additional file 1). The proposed scoping review will be reported in accordance with the reporting guidance provided in the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) extension for Scoping Reviews (PRISMA-ScR) [35].

Step 1: Defining the research questions

We are aware of heterogeneous or inconsistent use of the term “signal” [36], that signals of ADRs may be communicated by various stakeholders (see below for definition), that over time signals have been supported by a wide range of types of evidence, and that pharmacovigilance regulation is context-dependent. Defining key concepts thus becomes essential, beginning with the elemental components of a signal: one or more medicinal product (see [37]) and one or more suspected adverse drug reaction (see [38]). Crucially, we understand the concept “signal of ADR”, its subtypes (i.e. indeterminate, verified, refuted), and that of “SDRs” as previously defined [1]. These definitions, however, would commit us to determine whether any form of evidence, even single reports of ADRs, would command attention (“whose?”) or regulatory action (“of what type?”); making such judgments for every published and unpublished record would result in an unmanageable amount of work. We further acknowledge the subjectivity that comes into play in assessing some evidence as signal and that our judgments may differ from those of study authors, introducing bias. We therefore decided to consider as signals any findings that are merely described as such (e.g. “these results should be considered as a signal” or “the association between “drug A” and “ADR B” is a signal”) without distinguishing subtypes: following-up signals to ascertain any subsequent verificatory action would entail substantial additional work. We consider SDRs, i.e. results of disproportionality analyses (e.g. proportional reporting ratio (PRR), information component), those that exceed thresholds for statistical significance set by the study authors [28, 39, 40].

Pharmacovigilance, as defined by the European Union directive 2010/84/EU [41], regulation 1235/2010 [42], Good Vigilance Practices – Module IX [7], and the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use, efficacy guidelines E2A to E2F [43], will serve as references to contextualize pharmacovigilance in a global landscape.

We consider “stakeholders” to be healthcare practitioners and patients, national and international regulatory agencies or authorities, national and regional pharmacovigilance centres, non-governmental organizations, research institutions, academics, lawyers, and marketing authorization holders (i.e. pharmaceutical companies).

Based on these definitions, the questions this review will address are as follows:

  1. 1)

    What levels of evidence support signals of adverse drug reactions, and SDRs, communicated by drug regulatory authorities or other pharmacovigilance stakeholders?

  2. 2)

    What are the similarities and differences between the levels of evidence supporting signals from different stakeholders?

  3. 3)

    Have the levels of evidence changed over time?

  4. 4)

    What is the interval between a signal and the first report of the corresponding suspected adverse drug reaction?

  5. 5)

    Has this interval shortened or increased over the years?

  6. 6)

    In the case of signals supported by reports of adverse drug reactions, what features of the reports have led authors to signal suspected drug-related harms?

Step 2: Identify relevant articles

To ensure that searches in electronic databases would retrieve articles relevant to the research questions, we consulted a medical librarian. We initially ran pilot searches based on keywords in the list of the included references of [1] and iteratively refined the queries in line with the published literature on search strategies for adverse effects [44, 45]. Moreover, since signals may be unpublished, we intend to carry out grey literature screening, which will involve hand searches of selected bulletins. The librarian provided suggestions on how to rigorously query grey literature search engines.

Relevant articles will be retrieved as follows:

We shall search MEDLINE (via PubMed, 1946–present), EMBASE (1974–present), PsycINFO (1806–present), and Web of Science (indexes SCI-EXPANDED, CPCI-S, 1945–present) for signals published by organizations involved in pharmacovigilance and/or independent research groups (e.g. regional pharmacovigilance centres or academic departments) and pharmaceutical companies. We shall use Web of Science to perform backward citation screening; if we cannot use Web of Science, we will use Google Scholar.

We shall search the English language websites of 35 regulatory authorities or organizations involved in the World Health Organization (WHO) Programme for International Drug Monitoring, using the websites’ search engines, to identify communicated signals, expanding the list of countries compiled by Onakpoya et al. [22] where possible. If signals are not publicly available in a country, we shall forward requests to the competent authority. If the evidence used to support signals is not publicly available, we shall issue freedom of information requests (where regulated) for assessment reports or equivalent documents.

We shall hand-search the Uppsala Monitoring Centre (UMC)’s SIGNAL Document, the WHO Pharmaceuticals Newsletter, the Full List of WHO Medical Product Alerts, the WHO Drug Information, Australian Prescriber, Prescrire International, and other drug bulletins available in English via the International Society of Drug Bulletins’ index for relevant records.

We shall query Google Scholar in incognito browser tabs. We shall evaluate the results as long as 20 results (two consecutive pages) yield at least one useful article.

We shall also search OpenGrey and GreyNet International.

We shall use VigiBase [46], to obtain additional information on the year in which suspected ADRs were first reported.

The planned data lock point for the retrieval of all records is the end of August 2020.

A draft search strategy for PubMed and Google Scholar and sources of grey literature are provided as Additional file 2.

Step 3: Study selection

Since one of our goals is to better understand the evidence underpinning signals, we want to be as comprehensive as possible, while being able to remove irrelevant publications. Thus, we will not constrain by study design, language, or specific population: primary or secondary research using qualitative or quantitative methods will be eligible. We shall consider as eligible a priori disproportionality analyses [28, 39, 40], as these are exclusive to the detection of associations between medicinal products and ADRs.

Clearly, these criteria are insufficient for discarding all irrelevant publications, and we may obtain a large list of references. We shall therefore stipulate that any retrieved study should concern patients (i.e. excluding simulations and animal studies). We have therefore devised criteria for exclusion. First, descriptive, experimental, and observational study designs that are not unequivocally conducted to detect signals, even though the original authors may have described their findings as “signals”. An explicit description of the findings as “signals” in the abstract and in the full text will be necessary for eligibility. Secondly, different thresholds for significance may be used in disproportionality analyses, e.g. PRR ≥ 2 with Σ2 ≥ 4 with or without a minimum of three reports of ADRs. To account for this, we shall not apply standard thresholds, but we will require the presence of a threshold for significance, whether referenced or in the full text.

These criteria preserve the views of the original authors, minimize biases at our end, and allow us to work with a manageable data set.

Eligibility criteria

Inclusion criteria

  1. 1-

    Disproportionality analyses [28, 39, 40] that concern one or more medicinal products and one ADR or more and report at least one statistically significant association as per the threshold for significance adopted by the study authors.

  2. 2-

    Systematic reviews with or without meta-analyses, randomized controlled trials, observational studies, or more recent methods for signal detection (e.g. self-controlled case series, active surveillance) as detailed in [39], in which the original authors explicitly described any detected associations as “signal” in the abstract or full text.

  3. 3-

    Qualitative appraisals of individual reports or series of reports of ADRs, by either structured or unstructured methods (e.g. Naranjo’s algorithm, global introspection, and other methods for causality assessment [17]), that have been indexed as “signals” (e.g. signals from the UMC) or in which the original authors explicitly described one or more findings as “signal” in the abstract or full text.

  4. 4-

    Government or agency reports, working papers, and drug bulletins, whose entries have been indexed as signals or which explicitly describe an association between one or more medicinal products/ADRs as “signal” (viz. European Medicines Agency [47]).

Documents in languages other than English will be included if the review team can translate them into English. Requests for translations will be issued to the publishing stakeholder if the review team cannot translate.

Exclusion criteria

  1. 1-

    Abstracts or papers for which the full texts are unavailable.

  2. 2-

    Documents that claim that the evidence does not support a signal or that no signal was detected.

  3. 3-

    Signals that are communicated in the same or similar form or substance more than once by the same stakeholder (only the first signal in chronological order will be considered, if no new information appears in subsequent reports).

  4. 4-

    Non-English documents and websites for which a translation is unavailable.

Selection process

All retrieved records will be imported into EndNote™ X8.2. We shall deduplicate records manually, based on the lists of authors, titles, abstracts, journal, and pagination.

Retrieved titles and abstracts will be reviewed by DS (first author) against the inclusion and exclusion criteria. IJO (last author) will cross-validate the findings. Disagreements will be resolved by arbitration by JKA (second author).

Step 4: Charting the data

Eligible records will be reviewed by full text by DS. IJO will cross-validate the findings. Disagreements will be resolved by arbitration by JKA. Data charting will follow the same process.

Descriptive statistics

We shall extract data to compute descriptive statistics in a form created in the Systematic Review Data Repository, adapted to scoping reviews [48]. Specifically, the tabs we shall use are “Publications” and “Design”. Data to be extracted will be publication year, country of origin, stakeholder, study design, type of evidence, metric for detection (in the case of SDRs), population involved, setting (or database for SDRs), eligibility criteria, possible confounding factors, brand name of the medicinal product(s) (where available) or active pharmaceutical ingredient(s), formulation, considered ADR(s), definition of signal, year of the case report of the suspected ADR that was first transmitted to VigiBase and that relates to the same medicinal product, and ADR in a considered signal and/or of occurrence of the ADR (fields: “FirstDateDatabase” or, if complete, “ReactionStartDate”).

Features of reports of ADRs

To extract the features of reports of ADRs that were considered as supporting putative causal relationships, we shall include a set of pre-determined codes in the extraction form as multiple-choice checkboxes, each followed by a free-text field. These are “Time-to-onset”, “Dose-response relationship”, “Positive de-challenge”, “Positive re-challenge”, “Site-specificity of ADR”, and “Population-specificity of ADR”. Quotations from the full texts will be inserted in the respective free-text field. DS has been involved in qualitative assessments of reports of ADRs, and the codes have been based on his experience.

The extraction form will include a free-text field “Other” to accommodate the inclusion of other unanticipated features of reports of ADRs supporting putative causal relationships. Quotations from the full texts will be inserted in the field when they do not fall in any of the pre-specified codes and will be iteratively assigned codes in the form itself.

Step 5: Collating, summarizing, and reporting the results

We shall adopt a narrative approach to present the findings; these will be summarized by descriptive statistics and graphical illustrations after having standardized terminologies across included studies (medicinal products and ADRs) and codes that relate to the unanticipated features of reports of ADRs.

Standardization of terminologies

We shall code medicinal products to substance level, using the WHO Drug Dictionaries (WHO-DD, latest version available at the start of the review), and, where possible, code adverse events at the Low-Level Term level of the hierarchy in the Medical Dictionary for Regulatory Activities® (MedDRA) (latest version available at the start of the review), (e.g. “increased mortality” and seriousness of adverse events cannot be coded in MedDRA).

We shall translate types of evidence to the Oxford Centre for Evidence-Based Medicine levels of evidence framework

Codes extracted from the form in step 4 will be harmonized in the case of variance.

Descriptive statistics

We shall calculate absolute frequencies of types of evidence in Microsoft Excel and group them by type of stakeholder and/or year of publication. Similarly, we shall calculate the absolute frequencies of codes and publishing stakeholders.

Calculation of the time to signal

The complete (dd-mm-yy) date on which an ADR occurred, as stated in reports of ADRs entered in VigiBase, will be used to calculate the time-to-signal, by accounting for the date on which the signal relating to the same combination of medicinal product and the adverse effect was communicated. If the date of occurrence of the ADR is unavailable or incomplete, we shall use the first date on which the report was entered into VigiBase instead (field: UMCCalculated FirstDateDatabase). This field is necessarily complete, and follow-up reports are reconciled to it in VigiBase. For signals communicated by multiple stakeholders, the earliest date of communication will be considered. Depending on available resources, we may calculate the time-to-signal, beginning from the date on which cumulative disproportionality reaches statistical significance for the information component (lower limit of the 95% CI > 0).

Study findings and dissemination

Data synthesis

We shall present frequencies of types of evidence grouped by stakeholders in tables. We shall use figures (e.g. histograms) to present changes in frequencies of types of evidence over intervals of time.

We shall present all signals in tabular format, as combinations of medicinal product/ADR, or as medicinal product/medicinal product/ADR in the case of drug-drug interactions, together with their year of publication, stakeholder, eligibility criteria, population, setting, confounding factors, year of first entry in VigiBase, year of occurrence of the ADR (where available), and definition of signal. When signals are supported by spontaneous reports of ADRs, the codes of features of the reports of the ADRs will be included. Each signal will be accompanied by the time to signal.

A separate table will include SDRs and will include the same fields as above, plus the metric for detection (the definition of signal will be replaced by the specified threshold for significance and the database will replace the field “setting”).

We shall illustrate the time to signal for each stakeholder, in timeline plots and pictures (if appropriate).

Illustrations will be created using Microsoft Excel.


The results of the scoping review will consist of an overview of the levels of evidence underpinning signals. We shall highlight any discrepancies across stakeholders in levels of evidence, times to signal, and the most frequent features of reports of ADRs used to support putative causal relationships. We shall contextualize our findings with other published records and report the limitations of the study and recommendations for research. Variations from the published protocol will be presented and amendments made to the registered version. The full review will be submitted to a peer-reviewed journal for publication.


To the best of the authors’ knowledge, this would be the first study to systematically chart the evidence in support of signals and draw international comparisons of its levels using a standardized framework. Previous characterizations of the evidence underpinning “initial safety signals” [11] have focussed, for example, on Drug Safety Communications published by the US Food and Drug Administration [49]. These postings concern “important drug safety issue(s)” with “potential to alter the benefit–risk analysis for a drug in such a way as to affect decisions about prescribing or taking the drug” [50]. The agency’s website, however, also lists “potential signals of serious risks” [51], that concerns “about an excess of adverse events compared to what would be expected to be associated with a product’s use” [52]. This indicates that resorting to a pre-established set of webpages may result in the omission of relevant records and may misrepresent the activities of those regulatory agencies who refer to “signal” using different expressions. The time from the first report of an ADR to the communication of a related signal will also be quantified and compared across different healthcare settings, accounting for unpublished reports of ADRs available in VigiBase—the largest database of this kind. The findings from this review will underscore research questions for future reviews.

We expect to retrieve publications that report large lists of SDRs. This will pose some challenges, particularly during data charting, as the electronic form we choose may be unsuitable to accommodate all of them. Should this be the case, we shall adopt a customized Microsoft Excel sheet.

We anticipate that when abstracts do not transparently report methods, we may discard relevant records. In fact, there are no guidelines for reporting signals or SDRs. Furthermore, our exclusion criteria may result in the omission of eligible records: considering as eligible only records whose findings are described as “signals” is an approach that is insensitive to the use of other expressions (e.g. “safety risk”). In principle, contact with original authors would disambiguate those cases in which other expressions have been used, but might render such a broad review infeasible. In the case of contact with regulatory agencies, however, we may ascertain whether communications of, for example, “safety concerns” or “important safety concerns” were considered signals, as long as they reply to our queries. It is possible that we may not receive replies.

Furthermore, some publications may use discontinued terminologies to describe ADRs, such as the Coding Symbols for a Thesaurus of Adverse Reaction Terms, which have not been officially reconciled with MedDRA. This may introduce discrepancies in coding terms across terminologies. This is less likely to occur for medicinal products, as WHO-DD are quite comprehensive. However, WHO-DD are insensitive to vaccines; for example, trivalent inactivated influenza vaccine would be coded to “influenza vaccine”. We therefore plan to make this clear in the limitations of the resulting publication. Finally, as concerns the time to signal, the date of entry in VigiBase may be later than the date of receipt of a report to a national pharmacovigilance centre, which may underestimate the time to signal. Between the occurrence of an ADR and the date of entry in VigiBase, there may be a gap ranging from months to several years, thus exaggerating the time to signal. In the interest of transparency, we shall report both dates for each retrieved signal. Also, ADRs may have never been reported to VigiBase in association with some medicinal products, which would impede calculations of the time to signal.

Availability of data and materials

The data extraction form with the extracted data will be made public via The full URL will be made available once the scoping review is published. Data used to calculate the time-to-signal will also be made available, since it will be included in the extraction form; the dates of onset of an ADR, or dates of reporting to VigiBase, will be available on request to the corresponding author.



Adverse drug reaction(s)


Medical Dictionary for Regulatory Activities®


Proportional reporting ratio


Signal(s) of disproportionate reporting


Uppsala Monitoring Centre


World Health Organization


WHO Drug Dictionaries


  1. Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf. 2009;32(2):99–110.

    Article  PubMed  Google Scholar 

  2. Council for International Organizations of Medical Sciences. Practical aspects of signal detection in pharmacovigilance: report of CIOMS Working Group VIII. June Raine, editor. Geneva: World Health Organization; 2010.

  3. Bombardier C, Laine L, Reicin A, Shapiro D, Burgos-Vargas R, Davis B, et al. Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. VIGOR Study Group. N Engl J Med. 2000;343(21):1520–8.

    Article  CAS  PubMed  Google Scholar 

  4. European Medicines Agency. Summary Assessment Report of the PhVWP: Domperidone - risk of cardiac disorders. 2011. Accessed: 26 Mar 2020.

  5. van Noord C, Dieleman JP, van Herpen G, Verhamme K, Sturkenboom CJM. Domperidone and ventricular arrhythmia or sudden cardiac death. Drug Saf. 2010;33(11):1003–14.

    Article  PubMed  Google Scholar 

  6. Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol. 1998;54(4):315–21.

    Article  CAS  PubMed  Google Scholar 

  7. European Medicines Agency. Guideline on good pharmacovigilance practices (GVP) Module IX – Signal management 2017. Accessed: 26 Mar 2020.

  8. Farcaş A, Măhălean A, Bulik NB, Leucuta D, Mogoșan C. New safety signals assessed by the Pharmacovigilance Risk Assessment Committee at EU level in 2014–2017. Expert Rev Clin Pharmacol. 2018;11(10):1045–51.

    Article  CAS  PubMed  Google Scholar 

  9. Insani WN, Pacurariu AC, Mantel-Teeuwisse AK, Gross-Martirosyan L. Characteristics of drugs safety signals that predict safety related product information update. Pharmacoepidemiol Drug Saf. 2018;27(7):789–96.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Lester J, Neyarapally GA, Lipowski E, Graham CF, Hall M, Dal PG. Evaluation of FDA safety-related drug label changes in 2010. Pharmacoepidemiol Drug Saf. 2013;22(3):302–5.

    Article  PubMed  Google Scholar 

  11. Tau N, Shochat T, Gafter-Gvili A, Tibau A, Amir E, Shepshelovich D. Association between data sources and US Food and Drug Administration Drug safety communications. JAMA Intern Med. 2019.

  12. Perry LT, Bhasale A, Fabbri A, Lexchin J, Puil L, Joarder M, et al. Comparative analysis of medicines safety advisories released by Australia, Canada, the United States, and the United Kingdom. JAMA Intern Med. 2019;179(7):982–4.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Zeitoun J-D, Lefèvre JH, Downing N, Bergeron H, Ross JS. Inconsistencies among European Union pharmaceutical regulator safety communications: a cross-country comparison. PLoS One. 2014;9(10):e109100.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Hill AB. The environment and disease: association or causation? Proc R Soc Med. 1965;58(5):295–300.

    CAS  PubMed  PubMed Central  Google Scholar 

  15. Naranjo CA, Busto U, Sellers EM, Sandor P, Ruiz I, Roberts EA, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981;30(2):239–45.

    Article  CAS  PubMed  Google Scholar 

  16. Kramer MS. Assessing causality of adverse drug reactions: global introspection and its limitations. Drug Inf J. 1986;20(4):433–7.

    Article  Google Scholar 

  17. Agbabiaka TB, Savovic J, Ernst E. Methods for causality assessment of adverse drug reactions: a systematic review. Drug Saf. 2008;31(1):21–37.

    Article  PubMed  Google Scholar 

  18. Gan TJ, White PF, Scuderi PE, Watcha MF, Kovac A. FDA “black box” warning regarding use of droperidol for postoperative nausea and vomiting: is it justified? Anesthesiology. 2002;97(1):287.

    Article  Google Scholar 

  19. Handelsman Y, Henry RR, Bloomgarden ZT, Dagogo-Jack S, DeFronzo RA, Einhorn D, et al. American association of clinical endocrinologists and American College of Endocrinology position statement on the association of SGLT-2 inhibitors and diabetic ketoacidosis. Endocr Pract. 2016;22(6):753–62.

    Article  PubMed  Google Scholar 

  20. Meyer-Massetti C, Cheng CM, Sharpe BA, Meier CR, Guglielmo BJ. The FDA extended warning for intravenous haloperidol and torsades de pointes: how should institutions respond? J Hosp Med. 2010;5(4):E8–16.

    Article  PubMed  Google Scholar 

  21. Schumock GT, Lee TA, Joo MJ, Valuck RJ, Stayner LT, Gibbons RD. Association between leukotriene-modifying agents and suicide. Drug Saf. 2011;34(7):533–44.

    Article  PubMed  Google Scholar 

  22. Onakpoya IJ, Heneghan CJ, Aronson JK. Post-marketing withdrawal of 462 medicinal products because of adverse drug reactions: a systematic review of the world literature. BMC medicine. 2016;14:10.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Hashiguchi M, Imai S, Uehara K, Maruyama J, Shimizu M, Mochizuki M. Factors affecting the timing of signal detection of adverse drug reactions. PLoS One. 2015;10(12):e0144263.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Giezen TJ, Mantel-Teeuwisse AK, Straus SM, Schellekens H, Leufkens HG, Egberts AC. Safety-related regulatory actions for biologicals approved in the United States and the European Union. JAMA. 2008;300(16):1887–96.

    Article  CAS  PubMed  Google Scholar 

  25. Ishiguro C, Misu T, Iwasa E, Izawa T. Analysis of safety-related regulatory actions by Japan’s pharmaceutical regulatory agency. Pharmacoepidemiol Drug Saf. 2017;26(11):1314–20.

    Article  CAS  PubMed  Google Scholar 

  26. Powers A, Cook GE. Potential safety signals and their significance. Arch Intern Med. 2012;172(1):72–3.

    Article  PubMed  Google Scholar 

  27. Sriphiromya P, Theeraroungchaisri A. An analysis of legal warnings after drug approval in Thailand. Regul Toxicol Pharmacol. 2015;71(1):108–13.

    Article  PubMed  Google Scholar 

  28. Caster O, Juhlin K, Watson S, Norén GN. Improved statistical signal detection in pharmacovigilance by combining multiple strength-of-evidence aspects in vigiRank. Drug Saf. 2014;37(8):617–28.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Bergvall T, Norén GN, Lindquist M. vigiGrade: a tool to identify well-documented individual case reports and highlight systematic data quality issues. Drug Saf. 2014;37(1):65–77.

    Article  PubMed  Google Scholar 

  30. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005;8(1):19–32.

    Article  Google Scholar 

  31. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implementation Science. 2010;5(1):69.

    Article  PubMed  Google Scholar 

  32. Colquhoun HL, Levac D, O'Brien KK, Straus S, Tricco AC, Perrier L, et al. Scoping reviews: time for clarity in definition, methods, and reporting. Journal of Clinical Epidemiology. 2014;67(12):1291–4.

    Article  PubMed  Google Scholar 

  33. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews. 2015;4(1):1.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ : British Medical Journal. 2015;349:g7647.

    Article  Google Scholar 

  35. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

    Article  PubMed  Google Scholar 

  36. Hauben M, Reich L. Communication of findings in pharmacovigilance: use of the term “signal” and the need for precision in its use. Eur J Clin Pharmacol. 2005;61(5):479–80.

    Article  PubMed  Google Scholar 

  37. European Commission. On the approximation of provisions laid down by law, regulation or administrative action relating to proprietary medicinal products. Official Journal of the European Communities. 1965. Accessed: 26 Mar 2020.

  38. Edwards IR, Aronson JK. Adverse drug reactions: definitions, diagnosis, and management. Lancet. 2000;356(9237):1255–9.

    Article  CAS  PubMed  Google Scholar 

  39. Arnaud M, Bégaud B, Thurin N, Moore N, Pariente A, Salvo F. Methods for safety signal detection in healthcare databases: a literature review. Expert opinion on drug safety. 2017;16(6):721–32.

    Article  PubMed  Google Scholar 

  40. van Puijenbroek EP, Bate A, Leufkens HG, Lindquist M, Orre R, Egberts AC. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11(1):3–10.

    Article  CAS  PubMed  Google Scholar 

  41. European Commission. Directive 2010/84/EU of the European Parliament and of the Council of 15 December 2010. Eur-Lex. 2010. Accessed: 17 Jul 2020.

  42. European Commission. Regulation (EU) No 1235/2010 of the European Parliament and of the Council of 15 December 2010. Eur-Lex. 2010. Accessed: 17 Jul 2020.

  43. International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. Efficacy guidelines. 2020. Accessed: 26 Mar 2020.

  44. Golder S, Loke YK. The performance of adverse effects search filters in MEDLINE and EMBASE. Health Info Libr J. 2012;29(2):141–51.

    Article  PubMed  Google Scholar 

  45. Golder S, McIntosh HM, Duffy S, Glanville J. Developing efficient search strategies to identify reports of adverse effects in MEDLINE and EMBASE. Health Info Libr J. 2006;23(1):3–12.

    Article  PubMed  Google Scholar 

  46. Lindquist M. VigiBase, the WHO Global ICSR Database System: basic facts. Drug Inf J. 2008;42(5):409–19.

    Article  Google Scholar 

  47. European Medicines Agency. List of safety signals discussed since September 2012. 2020. Rev 1. Accessed: 27 Jun 2020.

  48. Systematic Review Data Repository. Accessed: 28 Jun 2020.

  49. United States Food and Drug Administration. Drug safety communications. 2020. Accessed: 26/03/2020.

  50. U.S. Food and Drug Administration. Guidance drug safety information – FDA’s communication to the public. 2015. Accessed: 26 Mar 2020.

  51. U.S. Food and Drug Administration. Potential signals of serious risks/new safety information identified from the FDA Adverse Event Reporting System (FAERS). 2020. Accessed: 26 Mar 2020.

  52. U.S. Food and Drug Administration. Guidance for industry - good pharmacovigilance practices and pharmacoepidemiologic assessment. 2005. Accessed: 26 Mar 2020.

Download references


We thank Nia Roberts, outreach librarian at the Bodleian Library of Oxford, for substantial contributions to the design of the search strategy and its adaptation to the electronic databases.


This protocol is part of Daniele Sartori’s DPhil in Evidence-Based Medicine, at the University of Oxford (Nuffield Departments of Primary Care and of Continuing Education), UK. The DPhil is fully funded by UMC. UMC has suggested some aspects, such as capturing the motivations that justify putative causal associations between medicinal products and adverse events. UMC manages part of the data that will be used to conduct this study, namely VigiBase, and the archives of the Signal Document. The Conflict of Interest section reiterates Daniele Sartori’s involvement in analysing this part of the data and writing of the resulting manuscripts.

Author information

Authors and Affiliations



DS designed the study, with input from IJO and JKA. NR substantially aided DS in the design of the search strategy and its adaptation to multiple databases. This manuscript was written by DS, IJO, and JKA. The authors read and approved the final manuscript.

Authors’ information

DS is a research pharmacist at UMC and reading for a DPhil in Evidence-Based Health Care at the University of Oxford, Nuffield Departments of Primary Care and Continuing Education, Oxford, UK.

JKA is a physician and clinical pharmacologist working at the Centre for Evidence-Based Medicine in the Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK.

IJO is an MD, with an MSc and DPhil in Evidence-Based Health Care from the University of Oxford, Oxford, UK.

Corresponding author

Correspondence to Daniele Sartori.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

JKA has written papers on adverse drug reactions in peer-reviewed journals and has received royalties from textbooks that he has edited or co-edited; he has often acted as an expert witness in cases involving adverse drug reactions, including opioids, most often in Coroners’ courts. DS has authored signals published on drug bulletins, as abstracts or posters, and received funding for doctoral studies from UMC. Part of the material expected to be reviewed has been published in peer-reviewed journals, drug bulletins, and abstracts or posters by other researchers presently or formerly belonging to this foundation. IJO has no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

PRISMA-P 2015 Checklist

Additional file 2.

Search strategy

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sartori, D., Aronson, J.K. & Onakpoya, I.J. Signals of adverse drug reactions communicated by pharmacovigilance stakeholders: protocol for a scoping review of the global literature. Syst Rev 9, 180 (2020).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: