- Open Access
- Open Peer Review
Clinical reminder alert fatigue in healthcare: a systematic literature review protocol using qualitative evidence
Systematic Reviewsvolume 6, Article number: 255 (2017)
Integrated reminders within clinical systems have become more prevalent due to the use of electronic health records and evidence demonstrating an increase in compliance within practice. Clinical reminders are assessed for effectiveness on an individual basis, rather than in combination with existing prompts for other conditions. The growing number of prompts may be counter-productive as healthcare professionals are increasingly suffering from “reminder fatigue” meaning many reminders are ignored. This work will review the qualitative evidence to identify barriers and enablers of existing prompts found within computerised decision support systems. Our focus will be on primary care where clinicians have to negotiate a plethora of reminders as they deal with increasingly complex patients and sophisticated treatment regimes. The review will provide a greater understanding of existing systems and the way clinicians interact with them to inform the development of more effective and targeted clinical reminders.
A comprehensive search using piloted terms will be used to identify relevant literature from 1960 (or commencement of database) to 2017. MEDLINE, MEDLINE In Process, EMBASE, HMIC, PsycINFO, CDSR DARE, HTA, CINAHL and CPCI, will be searched, as well as grey literature and references and citations of included papers. Manuscripts will be assessed for eligibility, bias and quality using the CASP tool with narrative data being included and questionnaire based studies excluded. Inductive thematic analysis will be performed in order to produce a conceptual framework defining the key barriers around integrated clinical reminders.
Indications of alert and reminder fatigue are found throughout the current literature. However, this has not been fully investigated using a robust qualitative approach, particularly in a rapidly growing body of evidence. This review will aid people forming new clinical systems so that alerts can be incorporated appropriately.
Systematic review registration
Implementation of desirable clinical behaviours, such as compliance with national guidelines, has taken a variety of forms within healthcare [1, 2]. One popular intervention is the use of an integrated electronic reminder or prompt , often taking the form of a ‘pop-up box’ within the clinical health record system [4, 5]. This type of prompt, usually delivered at the point of care, has two broad functions; reminding the user to perform a task, or to alert them about the potential consequences of not performing a task .
Approximately 90% of all patient consultations in the UK occur in primary care, equating to some 360 million appointments per year [7, 8]. The treatment of the majority of patients relies on guidelines designed to treat single diseases . However, current estimates indicate that 2.9 million people in the UK will be diagnosed as having two or more chronic conditions by 2018 , leading to a rise in the number of reminders for each patient, particularly where treatments are combined [11, 12]. These clinical reminders are used to influence clinician’s behaviour in a number of aspects of care including prescribing . In exploring the benefits of clinical support systems, a number of reviews have reported broadly positive effects. However, frequently these alerts are evaluated individually rather than in a cumulative fashion; furthermore, there has been a dramatic rise in the number of prompts, or reminders. Many are integrated within electronic health records and should be activated at the point of care to promote shared decision-making [13,14,15]. However, the numbers of prompts combined with increasing time pressures in the National Health Service (NHS) is increasing the cognitive load within this working environment , to such an extent that these reminders are largely being disabled, thus negating their effectiveness . This failure to engage with these digital prompts has been termed ‘alert fatigue’. By manually disabling all prompts, regardless of content, there are implications for patient safety, such as those providing warnings over medication , but also the increased interruption to the patient consultation can lead to a potential increase in medicolegal risk .
The aim of this research is to identify the barriers and enablers in the use of alerts and reminders within clinical systems as reported by healthcare professionals. Our focus will be within primary care and the electronic alerts found in this setting, including but not limited to alerts for prescribing, practice performance and lifestyle advice. This will be achieved by conducting a systematic review of the qualitative evidence to form our own overarching themes relating to the perceptions and experiences of alerts within computerised clinical decision support systems (CCDSS) . We will form these themes using an inductive approach  to facilitate the design of future clinical systems to minimise alert fatigue.
Specifically, we will locate and synthesise all qualitative evidence on the use of research evidence on the experience of electronic alerts in primary care. This will include data on
Barriers and enablers associated with the appropriate use of electronic alerts including aspects of their design and their frequency of appearance.
The experience of providers in using these alerts to inform their clinical decision-making.
This review will use systematic methodology to synthesise qualitative research evidence and has been registered with the International Prospective Register of Systematic Reviews (PROSPERO, www.crd.york.ac.uk/PROSPERO, CRD42016029418). This protocol is reported in line with PRISMA-P guidelines (Additional file 1), and the review will be reported according to PRISMA guidelines .
Research question: what are the barriers and enablers around the use of alerts/clinical reminders by healthcare professionals?
Below we outline the criteria against which studies will be included or excluded from the review. In order to be included in the review, the study must meet the following criteria;
Types of studies
This review will only include qualitative studies that report primary data. We are defining qualitative studies as those that qualitative methods for data collection and data analysis. This definition has been used in several recent qualitative syntheses [23,24,25]. Data collection can include but not be limited to semi-structured interviews (regardless of the method of construct analysis) and focus groups. Data analysis can include but not be limited to thematic analysis, grounded theory approach and discourse analysis. We will not include studies where data was collected qualitatively but analysed quantitatively such as via word counts using descriptive statistics. There will be no restriction on the location of the research for study inclusion, and abstracts written in other languages will be translated to assess eligibility. Papers will be included from 1960 (or commencement of database) to capture all initiatives and lessons learnt from alerts during the integration of the electronic health record into the NHS.
Types of settings and participants
We will include any qualitative study that consists of the perspective or experience of providers using alerts or reminders either as a standalone study or as part of mixed methods design. Providers are defined as healthcare professionals (to include doctors of all specialities and number of years of training, nurses and allied health professionals). If there is any lack of clarity as to the role of individuals within the study, we will contact the original author to seek clarification.
To be eligible for inclusion, all studies must explore the use of electronic alerts or reminders in primary care. This alert or reminder is being used either within a study or as part of routine clinical care. The outcomes are themes on barriers and enablers to adherence to alerts or reminders these can include factors external to the nature of the alert, e.g. characteristics of the provider the primary care environment or patient. Outcomes that will also be included relate to factors inherent to the alert such as its design, the content of text or the frequency with which it is displayed.
A comprehensive search strategy devised by SB (Information Specialist) and tested by RB (Health Services Researcher) will be used to identify relevant literature. A combination of free-text and index terms will be used in the following databases from 1960 (or commencement of database) to present: MEDLINE (Ovid), MEDLINE In Process (Ovid), EMBASE (Ovid), Health Management Information Consortium (HMIC) (Ovid), PsycINFO (Ovid), Cochrane Library (Wiley) CDSR (Cochrane Database of Systematic Reviews), DARE (Database of Abstracts of Reviews of Effects) and HTA (Health |Technology Assessment Database), Cumulative Index of Nursing and Allied Health (CINAHL) (EBSCO) and Conference Proceedings Citation Index (CPCI) via Web of Science.
The strategy will aim to combine sensitivity of free-text terms, truncation and spelling variations with the precision provided by controlled vocabulary (index terms). Search terms will be tailored to each database, grouped into three categories and combined using Boolean operators. These categories include terms for alerts and reminders, healthcare professionals and attitudes or challenges. Searches in the Ovid platform will where appropriate use the “qualitative (maximises specificity)” filter . A sample search for MEDLINE is included in Additional file 2 . There will be a search for grey literature including the OpenGrey database, and the websites of CCDSS suppliers (e.g. EMISHealth (https://www.emishealth.com) and the Phoenix Partnership (SystmOne https://www.tpp-uk.com/products/systmone)). Reference lists of all included articles will be checked for further relevant studies as will studies that cite included articles (using the Science Citation Index via Web of Science).
Data collection and analysis
In this section, we describe the planned methods for selecting studies and extracting and managing data. We also describe the means by which we will assess the quality of each study included in the review and how we will analyse and present the review findings.
Search results will be entered into Mendeley Reference Management Software (V1.17.6) where duplicate entries will be removed automatically by an in built algorithm supported by manual checking.
Study selection will be undertaken by two reviewers independently.
Titles and abstracts will be screened with reference to the eligibility criteria to remove all ineligible articles. Translation of abstracts will be sought if required, prior to eligibility assessment. The full text of articles remaining will be sought and considered for inclusion in the review using the full eligibility criteria. Part translations of articles not in English will be undertaken to facilitate this process. Disagreements between reviewers will be resolved through discussion and with referral to a third reviewer if required. Articles excluded from the review at the full-text stage will be noted on the data extraction form as will the eligibility criteria these did not satisfy.
Analysis will be undertaken using an iterative approach between reviewer one and two to form a pathway of alert usage, with the barriers and enablers marked from the data. Overarching themes from this data will then be established, and this will form the basis of a conceptual framework. Both reviewers will keep the context of the different healthcare systems in mind during this process, and if it is found that some barriers are more applicable to a specific context, for example, healthcare free at the point of access versus insurance based, this will be discussed in the final manuscript.
Critical appraisal, quality of reporting and data extraction
The CASP Qualitative Checklist (Critical Appraisal Skills Programme)  will be used to assess credibility, transferability, dependability and confirmability within the selected qualitative manuscripts at the study level. Where appropriate, we will also use the Standards for Reporting Qualitative Research . Data from duplicate publications of the same study will be combined rather than choose to assess just one of the publications. Both reviewer one and two will extract data independently using the CASP tool and if necessary, we will approach individual investigators to confirm data and methodology.
Because of the difficulties inherent in assessing every aspect of qualitative, work we will not exclude any based on the quality of their reporting  instead we will use a post-synthesis sensitivity analysis as recommended by Carroll et al. . This will allow us to determine whether any of our findings are based on a study of low quality as determined by the review team.
Data extraction and analysis
Results will be taken from the included papers, including appendices where appropriate, and will be imported into analysis software (NVivo v11). Inductive thematic analysis will then be performed to form overarching third order themes. To enable this, we will use the three stages described by Thomas and Harden  in the thematic synthesis of qualitative research. The first is the coding of the findings of the primary studies. The second is the categorisation of these codes into descriptive themes. The third is development of analytical themes to describe the themes that have emerged in the second phase. The approach we adopt will be inductive, i.e. themes emerge from the data through repeated examination and comparison. The emerging descriptive themes will be analysed in consideration of the contextual factors, i.e. healthcare context, setting and reminder usage within clinical decision systems to assess if these contextual factors have any impact upon the ensuing analytical themes. The findings will be verified by utilising independent coding by two reviewers, the triangulation of these codes and iterative discussions amongst all reviewers of the coding framework at each of the three phases of the analysis.
Presentation of findings and reporting methods
The review will describe participant and setting characteristics, data collection and analysis methods. The findings of the primary papers will be summarised in tabular form describing the key characteristics of each. In addition, we will describe each paper narratively. The review findings will be classified into key themes as informed by the analysis. The review’s findings will also be summarised visually in a proposed conceptual framework explaining the relationship between the key factors influencing the efficacy of electronic alerts.
The protocol was developed and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) [22, 32] (Additional file 1). The review methods and results will be reported according to the ENTREQ (enhanced transparency in reporting the synthesis of qualitative research statement) for reporting synthesis of qualitative studies . The final literature searches will be reported using the STARLITE (sampling strategy, type of study, approaches, range of years, limits, inclusions and exclusions, terms used, electronic searches) .
Current evidence suggests that alerts have a range of effectiveness [34,35,36,37,38] yet an understanding of the factors that influence discrepancies inefficacy is lacking. There have been several reviews around the use of clinical support tools [34, 35, 39] though there are few high quality reviews which have ultimately attempted to construct a conceptual framework from the qualitative data. Similarly lacking is robust evidence on how the characteristics of alerts and the way they are deployed can impact on the phenomenon of alert fatigue in primary care. Therefore, our review will add considerably to the existing evidence base. The systematic nature of our approach and the comprehensive acquisition of evidence being assessed will provide further insights into how alerts can be altered to improve practitioner and patient outcomes and help explain the heterogeneity in effectiveness and offer suggestions that will increase utility and usability of alerts within CCDSS.
At the time of submission, full searches had not been run as per the strategy above and scoping work had been undertaken.
Critical Appraisal Skills Programme
Computerised clinical decision support systems
Cumulative Index to Nursing and Allied Health Literature
Conference Proceedings Citation Index
Enhanced transparency in reporting the synthesis of qualitative research statement
Health Management Information Consortium
National Health Service
National Institute for Health and Care Excellence
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
International Prospective Register of Systematic Reviews
Randomised controlled trial
Sampling strategy, type of study, approaches, range of years, limits, inclusion and exclusions, terms used, electronic sources
Grimshaw J, Eccles M, Tetroe J. Implementing clinical guidelines: current evidence and future implications. J Contin Educ Heal Prof. [Internet]. 2004; 24 Suppl 1:S31-S37. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15712775 [cited 2016 Apr 5].
Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8:iii–v. 1-72. Available from: http://www.ncbi.nlm.nih.gov/pubmed/14960256 [cited 2015 Nov 25]
Dexheimer JW, Talbot TR, Sanders DL, Rosenbloom ST, Aronsky D. Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials. J Am Med Inform Assoc. 2008;15:311–20. Available from: http://www.sciencedirect.com/science/article/pii/S1067502708000170 [cited 2016 5]
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22696318 [cited 2015 Jun 29]
Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2010:CD005470. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=4164371&tool=pmcentrez&rendertype=abstract [cited 2015 Nov 25]
Brokel JM, Schwichtenberg TJ, Wakefield D, Ward M, Shaw M, Kramer JM. Evaluating clinical decision support rules as an intervention in clinician workflows with technology. CIN Comput Informatics, Nurs. 2011;29:36–42.
Roland M, Everington S. Tackling the crisis in general practice. BMJ. 2016;352:i942. Available from: http://www.bmj.com/content/352/bmj.i942 [cited 2016 19]
(mr) Web Master UK. Primary care [Internet]. (mr) Web Master, United Kingdom; [cited 2016 Feb 12]. Available from: http://www.hscic.gov.uk/primary-care.
Hughes LD, MET MM, Guthrie B. Guidelines for people not for diseases: the challenges of applying UK clinical guidelines to people with multimorbidity. Age Ageing. 42:62–9. Available from: http://ageing.oxfordjournals.org/content/42/1/62.full Oxford University Press; 2013 [cited 2016 12]
Long Term Conditions Compendium of Information: Third Edition - Publications - GOV.UK [Internet]. [cited 2016 Feb 23]. Available from: https://www.gov.uk/government/publications/long-term-conditions-compendium-of-information-third-edition
Grace A, Mahony C, O’Donoghue J, Heffernan T, Molony D, Carroll T. Evaluating the effectiveness of clinical decision support systems: the case of multimorbidity care. J Decis Syst. 2013;22
Schedlbauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc. 2009;16:531–8. Available from: http://jamia.oxfordjournals.org/content/16/4/531.abstract [cited 2016 5]
Litvin CB, Davis KS, Moran WP, Iverson PJ, et al. The Use of Clinical Decision-Support Tools to Facilitate Geriatric Education. J Am Geriatr Soc. 2012:1145–9. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=ovftm&NEWS=N&AN=00004495-201206000-00022
Trevana L, McCaffery K, Salkeld G, Glasziou P, Mar C del, Doust J, et al. Clinical decision-making tools: how effective are they in improving the quality of health care? Fac. Heal. Sci. Med. Publ. 2014. Available from: http://epublications.bond.edu.au/hsm_pubs/861 [cited 2016 Feb 12].
Elwyn G, Scholl I, Tietbohl C, Mann M, Edwards AGK, Clay C, et al. “Many miles to go …”: a systematic review of the implementation of patient decision support interventions into routine clinical practice. BMC Med Inform Decis Mak. 2013;13 Suppl 2:S14. Available from: http://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/1472-6947-13-S2-S14 [cited 2015 Dec 29]
Ariza F, Kalra D, Potts HW. How do clinical information systems affect the cognitive demands of general practitioners?: usability study with a focus on cognitive workload. J Innov Heal Informatics. 2015;22:379–90. Available from: http://hijournal.bcs.org/index.php/jhi/article/view/85/857 [cited 2016 Feb 1]
Moxey A, Robertson J, Newby D, Hains I, Williamson M, Pearson S-A. Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc. 2010;17:25–33. Available from: http://jamia.oxfordjournals.org/content/17/1/25.abstract [cited 2016 Apr 5]
Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163:2625–31. Available from: http://archinte.jamanetwork.com/article.aspx?articleid=216364&resultclick=1 [cited 2016 Apr 5]
Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce “alert fatigue” while still minimizing the risk of litigation. Health Aff (Millwood). 2011;30:2310–7. Available from: http://content.healthaffairs.org/content/30/12/2310.short [cited 2016 Feb 19]
Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18616818 [cited 2016 Nov 24]
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101. Available from: http://www.tandfonline.com/doi/abs/10.1191/1478088706qp063oa [cited 2016 Dec 12]
PRISMA [Internet]. [cited 2016 Feb 17]. Available from: http://www.prisma-statement.org/.
Verboom B, Montgomery P, Bennett S. What factors affect evidence-informed policymaking in public health? Protocol for a systematic review of qualitative evidence using thematic synthesis. Syst Rev. 2016;5:61. Available from: http://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-016-0240-6 [cited 2017 Feb 17]
Noyes J, Popay J. Directly observed therapy and tuberculosis: how can a systematic review of qualitative research contribute to improving services? A qualitative meta-synthesis. J Adv Nurs. 2007;57:227–43. Available from: http://doi.wiley.com/10.1111/j.1365-2648.2006.04092.x [cited 2017 Feb 17]
Glenton C, Colvin CJ, Carlsen B, Swartz A, Lewin S, Noyes J, et al. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev. 2013; Available from: http://doi.wiley.com/10.1002/14651858.CD010414.pub2 [cited 2017 Feb 17]
Wong SS-L, Wilczynski NL, Haynes RB, Hedges Team. Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. Stud Health Technol Inform. 2004;107:311–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15360825 [cited 2017 Feb 1]
Critical Appraisal Skills Programme. CASP Qualitative Checklist [online] [Internet]. 2017 [cited 2017 Oct 23]. Available from: http://www.casp-uk.net/checklists.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups.
Dixon-Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative research. Qual Saf Health Care. 2004;13:223–5. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15175495 [cited 2017 Feb 17]
Carroll C, Booth A, Leaviss J, Rick J. “Best fit” framework synthesis: refining the method. BMC Med Res Methodol. 2013;13:37. Available from: http://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-13-37 [cited 2017 Feb 17]
Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59. doi:10.1186/1471-2288-9-59.
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 2015;4:1. Available from: http://systematicreviewsjournal.biomedcentral.com/articles/10.1186/2046-4053-4-1 [cited 2015 Jan 3]
Booth A. "brimful of STARLITE": toward standards for reporting literature searches. J Med Libr Assoc. 2006;94:421–9. e205. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17082834 [cited 2017 Feb 17]
Moja L, Kwag KH, Lytras T, Bertizzolo L, Brandt L, Pecoraro V, et al. Effectiveness of computerized decision support systems linked to electronic health records: a systematic review and meta-analysis. Am J Public Health. 2014;104:e12–22. Available from: http://ajph.aphapublications.org/doi/abs/10.2105/AJPH.2014.302164 [cited 2016 Feb 13]
Ranji SR, Rennke S, Wachter RM. Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review. BMJ Qual Saf. 2014;23:773–80. Available from: http://qualitysafety.bmj.com/content/23/9/773 [cited 2015 Nov 1]
Nuckols TK, Smith-Spangler C, Morton SC, Asch SM, Patel VM, Anderson LJ, et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst. Rev. 2014;3:56. Available from: http://systematicreviewsjournal.biomedcentral.com/articles/10.1186/2046-4053-3-56 [cited 2015 Dec 21]
Georgiou A, Prgomet M, Paoloni R, Creswick N, Hordern A, Walter S, et al. The effect of computerized provider order entry systems on clinical care and work processes in emergency departments: a systematic review of the quantitative literature. Ann Emerg Med. 2013;61:644–53. e16. Available from: http://www.annemergmed.com/article/S0196064413001005/fulltext [cited 2016 Feb 17]
Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10:523–30. Available from: http://jamia.oxfordjournals.org/content/10/6/523.abstract [cited 2015 Jul 13]
Cresswell K, Majeed A, Bates DW, Sheikh A. Computerised decision support systems for healthcare professionals: an interpretative review. Inf Prim Care. 2012:115–28. Available from: http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=ovftm&NEWS=N&AN=00151298-201220020-00006 [cited 2015 Dec 10]
This work forms part of the grant ‘Automated Conflict Resolution in Clinical Pathways’ (EP/M014401/1) which has been funded by the Engineering and Physical Sciences Research Council (EPSRC).
Availability of data and materials
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
- Computerised clinical decision support systems
- Clinical reminders
- Alert fatigue
- Clinical reminder fatigue
- Systematic review