- Open Access
- Open Peer Review
Measured and perceived effects of audit and feedback on nursing performance: a mixed methods systematic review protocol
Systematic Reviewsvolume 8, Article number: 38 (2019)
The use of audit and feedback (A&F) interventions in health care has been demonstrated to generally be effective on medical teams. However, literature suggests that the response of nurses to this type of intervention may differ from that of other types of health professionals, in relation to their roles, power, and to the configuration of nursing care activities. To our knowledge, no review has been conducted on A&F interventions with nurses. The objective of this systematic review is to examine the evidence of measured and perceived effects of A&F interventions on nurses’ performance.
A mixed methods systematic review design with thematic and narrative synthesis is used. Studies reporting quantitative and qualitative data on the effects of A&F interventions specific to nursing care are considered for inclusion. Studies will be appraised for quality using the Mixed Methods Appraisal Tool. Quantitative and qualitative data will be summarized in narrative and tabular form and will be synthetized using a segregated methodologies approach.
Results will describe the characteristics of A&F with nurses, as well as the measured and perceived effects specific to nursing care. The associations between the characteristics and the effects as well as the concordance between measured and perceived effects will be presented. We anticipate that combining the evidence from qualitative and quantitative studies will allow us to provide relevant insight that can inform the design of better suited A&F interventions for nurses. Audit and feedback interventions demonstrate potential for improving the performance of nursing care. As their effectiveness varies greatly depending on the context and the professionals involved, a better understanding of the associations between its characteristics and the measured and perceived effects is valuable for improving the effectiveness of A&F.
Systematic review registration
Scientific literature on quality improvement in healthcare highlights the effectiveness of audit and feedback (A&F) [1,2,3,4,5]. A&F is an intervention comprising of two components: (1) evaluation of the performance (audit) and (2) feedback to the professionals. Most of the literature pertaining to A&F involves physicians [1, 5,6,7]. However, some authors suggest that the response of nurses to this type of intervention may differ from that of other types of health professionals  in relation to their roles, power, and the configuration of nursing care activities [9,10,11]. Since nurses often work in teams, quality improvement involves changes both at individual and collective levels. Collective improvements, requiring communication, cohesion, and coordination within the nursing team [11, 12], make interventions with this group of professionals particularly challenging.
A&F should focus on elements of practice over which professionals perceive to have control and to be accountable for [1, 4], which has been suggested to be a current limitation in A&F interventions involving nurses [9,10,11]. Although many studies report the development and use of nursing-sensitive indicators, the access to measurable process indicators is currently lacking [13,14,15,16]. The use of process indicators, as they are more actionable and proximal to nursing care activities, could result in higher improvement rates of nursing performance . The nature of the indicators reported to nurses could therefore influence the overall effectiveness of A&F. In order to design A&F interventions that are suitable for nursing activities, it is relevant to gather the current state of evidence on A&F targeting nurses.
To our knowledge, no systematic review has been conducted on A&F interventions with nurses. Mixed methods systematic reviews are a method of synthesizing knowledge which aims to combine several types of data concerning the same subject in order to present an integrated synthesis of the results produced through different methodologies . A better understanding of how A&F unfolds in nursing practice settings could allow us to implement design elements according to the care settings and professionals involved. Given the paucity of literature on the effectiveness of A&F interventions with nurses  and the difficulty of quantitatively measuring nursing-specific effects [14, 18], it is relevant to study both the measured and perceived effects of this type of intervention. Performance, in the current paper, is addressed in a comprehensive way  and refers to dimensions of quality, effectiveness, and security.
Characteristics of A&F interventions
Although the efficacy of A&F interventions has been well demonstrated, it is well acknowledged that it has large variations depending on certain characteristics [1, 20, 21]. Some authors have identified A&F characteristics which are more likely to modify its delivery and would therefore explain those variations [1, 2]. These characteristics relate to the form, content, and frequency of interventions . Until now, no consensus has been established to provide best practices for developing A&F interventions. The effectiveness of A&F may also be influenced by, among other things, the type of professional, the context of the intervention, and levels of baseline performance .
The effects of A&F interventions are usually assessed using quantitative benchmarks in the form of indicators, which monitor the action around the dimensions of performance . As such, the structure-process-outcome triad developed by Donabedian  is a quality of care assessment framework designed to include all dimensions of performance of a health organization. To Donabedian , the structure component refers to all the elements relating to the framework in which care is provided, including both elements related to environment and those related to staff characteristics. The process component encompasses all the acts performed by the professional in their care delivery activities as well as the way these activities are performed. The outcome component refers not only to the effects of care on the health status of populations , but also to organizational outcomes, including the volume of care provided by a health facility . In this review, this triad will be used to extract and analyze data from quantitative studies.
Quantitative evaluation of these three categories of indicators requires access to data . In nursing, access to data that are both valid and reliable that allows nursing indicators’ measurement is an issue [18, 25,26,27]. At present, the measurable effects of A&F interventions on nursing performance have some limitations, such as variable access to reliable nursing-specific indicators [14,15,16]. Hence, it seems highly relevant to review the perceived effects associated with this type of intervention, which are generally evaluated through qualitative studies. Given the small amount of literature related to nurse-specific effects of AF , the inclusion of perceived effects is intended to provide a better understanding of how this type of intervention unfolds with nurses influences the perception of their performance. Murray et al.  proposed a framework for describing and evaluating the effects of implementing a complex intervention, such as A&F, within a health organization from the perspective of the professionals involved. We believe this framework provides a global and relevant perspective to better understand the actions of the professionals involved in the implementation of a complex intervention. According to this approach, the effects perceived by professionals when implementing an intervention in their practice relate to four areas. The first area, that of coherence, relates to the meaning and value that participants place on intervention . The second area, that of cognitive participation, refers to the commitment and motivation that the implementation of the intervention entails for participants. The third area, that of collective action, refers to the actions that intervention requires in participants’ practice changes, for example, in terms of time and training. The last area, the reflexive domain, refers to the perceived effects of changes in practice, for example, in improving the care provided . The assessment of these domains makes it possible to consider the context of implementation of the intervention and its effects on the response of the participants . This framework will be used in this review to ensure the systematic nature of the data extraction and analysis from qualitative studies.
Objective and research questions
The main objective of this review is to identify and appraise the existing evidence on the effects of A&F on nursing performance. It aims to answer the following questions:
What are the main features of A&F with nurses?
What are the measured effects of A&F on nurses’ performance?
What are the perceived effects of A&F on nurses’ performance?
What are the associations between the features of A&F and the measured and perceived effects?
What is the concordance between the measured and perceived effects of A&F with nurses?
Considering mixed methods reviews’ approaches are still emerging, we will conduct the data extraction, analysis, and synthesis using a various range of guidelines [17, 29,30,31,32,33]. The specific contributions and applications of these guidelines are presented in each section of the method. The Preferred Reporting Items for Systematic Reviews and Meta-Analyzes (PRISMA) grid  will be used as a reference for the preparation and writing of this review. This mixed methods systematic review is registered with the International Prospective Register of Systematic Reviews (PROSPERO) CRD42018104973.
The primary outcome will include any measured or perceived effect of the AF intervention on nursing-related dimensions of performance, such as quality and security of care and effectiveness. A measured effect is defined as any variable that is reported objectively and assessed through quantitative methods. A perceived effect is defined as any variable that is subjectively reported and assessed through qualitative methods.
Criteria have been developed to specifically address the research questions. Thus, studies that meet the following criteria will been included: (1) A&F intervention corresponding to a feedback of the performance to nurses over a given period of time , (2) focusing on an A&F intervention with nurses or health professionals including nurses and with specific effects on nurses, (3) describing characteristics of the A&F intervention, and (4) presenting an A&F intervention related to at least one effect related to a dimension of performance. The literature that will meet the following criteria will be excluded: (1) providing feedback to nursing students and (2) focusing on an A&F intervention targeting patients.
The strategy will be carried out in three stages. First, an initial search will be performed from CINAHL and MEDLINE to test the identified keywords and descriptors . Then, a second search will be carried out based on the keywords and descriptors identified in all the selected databases. Then, the list of references of identified studies will be analyzed to identify additional articles. Authors recognized as experts in the field will be contacted by email to obtain any additional relevant study related to the research topic . Online periodicals relevant to the area of interest will be identified and searched for additional and non-indexed articles in the databases . Publications in English and French will be considered with no time restriction. The following databases will be used: CINAHL, Cochrane Controlled Register of Trials, Embase, MEDLINE, PubMed, Scopus, Web of Science, PsycINFO, and Global Health. The ProQuest database will be considered for tracking unpublished studies. Quantitative studies will include randomized controlled trials, nonrandomized studies, and descriptive studies . Qualitative studies will include phenomenological studies, grounded theorization, ethnography, action research, case studies, and descriptive qualitative research . Mixed methods studies, if any, will be considered.
The following concepts will be sought: “feed back” OR “action planning” OR feedback OR audit OR audits AND nurs* AND performance OR quality OR effectiveness OR security OR efficacy. The following descriptors will be used: (MH “Feedback”) OR (MH “Audit”) OR (MH “Nursing Audit”) OR (MH “Clinical Audit”) AND (MH “Nursing Role”) OR (MH “Nurses +”) OR (MH “Nursing Care +”) AND (MH “Quality of Health Care”) OR (MH “Quality Improvement”) OR (MH “Clinical Effectiveness”) (see Additional file 1). Search equivalents terms in French will be sought.
Data screening and extraction process
The lead review author (ED) will upload the search results and remove duplicates using EndNoteX8 (Clarivate Analytics). Search results will then be imported into Covidence (https://www.covidence.org), an online systematic review software, to conduct screening, data extraction and quality assessment. Two independent reviewers (ED and JB) will then screen titles and abstracts and classify studies (1) possibly meeting the eligibility criteria, (2) not meeting the eligibility criteria. Studies possibly meeting the inclusion criteria will then be obtained in full-text format. Two review authors will independently complete the full-text screening process (ED and JB). Any disagreement will be discussed and resolved with a third reviewer (AD). A PRISMA flow-chart diagram  will be used to report study selection process. Data extraction will be performed by two independent reviewers (ED and JB). For all studies, a grid will be used to extract data on the design, the geographic location, the context, the participants, and the objective of the included studies.
For all studies, a table inspired by the questioning formula what, why, who, when, and how that Colquhoun et al.  used to classify the characteristics of AF interventions with medical teams will serve the same purpose in our review. Since the object of interest constitutes a heterogeneous field of research , we chose to proceed with the steps of extraction, analysis, and presentation of results using reference frameworks previously selected by the research team to ensure a systematic process. Evidence from quantitative studies will be classified in a systematic way following the Donabedian’s quality of care triad . Evidence from qualitative studies will be classified in a table inspired by Murray et al.’s framework . This framework will be used to identify themes and classify perceived effects. Data on measured and perceived effects will be extracted from the results and discussion sections of the studies. Data will include statistical analyses, verbatim quotations, and narrative descriptive summaries of results. The extraction will draw on the method proposed by Sandelowski et al.  in order to preserve the methodological context of the findings.
Quality assessment of included studies
The quality assessment of included studies will be carried out using the tool Mixed Methods Appraisal Tool (MMAT) . This tool was developed specifically to assess the quality of items with a quantitative, qualitative, or mixed methods design . The quantitative studies will be first categorized according to their characteristics and then assessed according to their respective characteristics, including the sampling strategy, the measuring instruments, and the response rate. For qualitative studies, the assessment criteria will include context, data sources, and data analysis. The evaluation criteria for the mixed methods studies will include the integration of the methods and the limitations presented. Two reviewers (ED and JB), for each of the criteria, will independently assign a score based on “yes,” “no,” “unspecified,” or “not applicable” responses. Any disagreement between the reviewers will be resolved by discussion or by requesting the assessment of a third reviewer (AD).
Given the expected heterogeneity of quantitative data, the evidence on measured effects will be reported in a narrative form following a thematic analysis  based on Donabedian’s quality of care triad . Qualitative data will be analyzed through the areas described in Murray et al.  framework. Data analysis of evidence from quantitative and qualitative studies will be conducted following the three-step narrative approach proposed by Popay et al. . This approach is suited for inclusion of a wide range of research designs that generate both quantitative and qualitative findings on the effects of an intervention . This approach encompasses (1) developing a preliminary synthesis, (2) exploring relationships in the data, and (3) assessing the robustness of the synthesis product . A primary synthesis describing the patterns of direction and size of the effects will be undertaken from the extraction stage and considering the context and characteristics of the included studies . This first synthesis will allow for an in-depth exploration of relationships between contexts, processes, and outcomes . Patterns across studies will be identified by the reviewers in order to synthesis how characteristics and contexts influence the effects of A&F. Following this in-depth synthesis, its robustness will be assessed by considering the methodological quality of the included studies based on the MMAT .
Based on the segregated methodologies proposed by The Joanna Briggs Institute [17, 30] and based on Sandelowski et al. , an individual synthesis of quantitative and qualitative data will be performed. In this approach, a clear distinction between quantitative and qualitative evidence is required prior to the mixed method synthesis . This first individual synthesis will be conducted following the approach proposed by Popay et al.  as previously described.
A mixed method synthesis including all the generated findings will be then be conducted in order to study (1) the association between the A&F features and the measured or perceived effects and (2) the concordance between the measured effects and the perceived effects of A&F interventions with nurses. The combined findings will either support, contradict, or add to the quantitative and qualitative evidence . This mixed method synthesis will follow a configuration approach [32, 39]. Using a configuration will allow to make connections between the evidence from both the quantitative and qualitative individual syntheses without merging them . In this review, quantitative data are used to inform measured effects while qualitative data are used to inform perceived effects. Considering both types of effects have different implications for nursing performance, the findings will be used to complement rather than confirm each other . Figure 1 presents the data analysis plan for this mixed method systematic review.
Assessing confidence in the evidence
An assessment of the confidence in the evidence will be conducted from the recommendations made by The Joanna Briggs Institute [40,41,42]. These recommendations have been formulated in a complementary approach to that of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool  and are based on the assessment of the coherence of the evidence according to the study specifications . Given the heterogeneity of approaches to evaluating the effects of A&F interventions  as well as the inclusion of qualitative studies, this approach is favored in order to make appropriate recommendations. The levels of evidence proposed by The Joanna Briggs Institute , in a similar approach to that of GRADE, include the feasibility, relevance, effectiveness, and meaningfulness of the intervention under study. The latter relates more particularly to the evidence assessment for qualitative studies .
The primary objective of this mixed methods systematic review is to gather evidence on the measured and perceived effects of A&F interventions on nursing performance. Although systematic reviews on measured effects of A&F have been conducted on professionals’ practice and health outcomes , no review has focused on gathering effects specific to nursing practice. Furthermore, no review has aimed to assess evidence from qualitative as well as quantitative studies. We anticipate that combining these findings to qualitative studies will be highly relevant in providing insight on nurses’ perceptions on A&F and therefore to provide recommendations that can improve its design. This review will be valuable to health professionals interested in designing better suited A&F interventions for nurses. The strengths and the limits of the review, as well as recommendations for the implementation of A&F in nursing care will be discussed. One of the main limitations of this review is the heterogeneity in the literature regarding A&F in nursing. We anticipate A&F studies will take place in many different health settings, as well as in various forms and on different topics. We address this potential limitation by using frameworks specific to each type of data. The heterogeneity of the evidence will also be considered for the recommendations made following the evidence assessment.
Audit and feedback
Grading of Recommendations Assessment, Development and Evaluation
Mixed methods appraisal tool
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, Ivers N, Grimshaw JM. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164(6):435–41.
Colquhoun H, Michie S, Sales A, Ivers N, Grimshaw JM, Carroll K, Chalifoux M, Eva K, Brehaut J. Reporting and design elements of audit and feedback interventions: a secondary review. BMJ Quality & Safety. 2016;26(1):54–60.
Ivers N, Grimshaw J. Reducing research waste with implementation laboratories. Lancet. 2016;388(10044):547–8.
Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, Grimshaw JM. No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014;9(1):14.
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259.
Larson EL, Patel SJ, Evans D, Saiman L. Feedback as a strategy to change behaviour: the devil is in the details. J Eval Clin Pract. 2013;19(2):230–4.
Lu CY, Vinci LM, Quinn MT, Wilkes AE, Chin MH, Peek ME. Using feedback to change primary care physician behavior. J Ambul Care Manage. 2015;38(2):118–24.
Giesbers APM, Schouteten RLJ, Poutsma E, van der Heijden BIJM, et van Achterberg T. Feedback provision, nurses’ well-being and quality improvement: towards a conceptual framework. J Nurs Manag. 2015;23(5):682–691. https://doi.org/10.1111/jonm.12196.
Cheater FM, Baker R, Reddish S, Spiers N, Wailoo A, Gillies C, Cawood C. Cluster randomized controlled trial of the effectiveness of audit and feedback and educational outreach on improving nursing practice and patient outcomes. Med Care. 2006;44(6):542–51.
Leeman J, Baernholdt M, Sandelowski M. Developing a theory-based taxonomy of methods for implementing change in practice. J Adv Nurs. 2007;58(2):191–200.
Holleman G, Poot E, Mintjes-de Groot J, van Achterberg T. The relevance of team characteristics and team directed strategies in the implementation of nursing innovations: a literature review. Int J Nurs Stud. 2009;46(9):1256–64.
Rahn DJ. Transformational teamwork: exploring the impact of nursing teamwork on nurse-sensitive quality indicators. J Nurs Care Qual. 2016;31(3):262–8.
Dubois CA, D'Amour D, Brault I, Dallaire C, Dery J, Duhoux, A, Zufferey A. Which priority indicators to use to evaluate nursing care performance? A discussion paper. J Adv Nurs. 2017;73(12):3154–67. https://doi.org/10.1111/jan.13373.
Heslop L, Lu S. Nursing-sensitive indicators: a concept analysis. J Adv Nurs. 2014;70(11):2469–82.
VanFosson CA, Jones TL, Yoder LH. Unfinished nursing care: an important performance measure for nursing care systems. Nurs Outlook. 2016;64(2):124–36.
Xiao S, Widger K, Tourangeau A, Berta W. Nursing process health care indicators: a scoping review of development methods. J Nurs Care Qual. 2017;32(1):32–9.
Joanna Briggs Institute. The Joanna Briggs Institute Reviewers’ Manual 2014. In: Methodology for JBI Mixed Methods Systematic Reviews. Adelaide: The Joanna Briggs Institute; 2014.
Dubois C-A, D’Amour D, Brault I, Dallaire C, Déry J, Duhoux A, Lavoie-Tremblay M, Mathieu L, Karemere H, Zufferey A. Indicateurs prioritaires pour évaluer la contribution infirmière à la qualité des soins : revue systématique des écrits. Montréal: Secrétariat international des infirmières et infirmiers de l’espace francophone; 2015.
Sicotte C, Champagne F, Contandriopoulos AP, Barnsley J, Beland F, Leggat SG, Denis JL, Bilodeau H, Langley A, Bremond M, et al. A conceptual framework for the analysis of health care organizations’ performance. Health Serv Manag Res. 1998;11(1):24–41.
Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, Sales A, Brehaut JC. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci. 2017;12(1):117.
Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res. 2005;5:50.
Donabedian A. The quality of care: how can it be assessed? J Am Med Assoc. 1988;260(12):1743–8.
Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care. 2001;13(6):475–80.
Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care. 2003;15(6):523–30.
Kurtzman ET, Dawson EM, Johnson JE. The current state of nursing performance measurement, public reporting, and value-based purchasing. Policy, Politics, & Nursing Practice. 2008;9(3):181–91.
Needleman J, Kurtzman ET, Kizer KW. Performance measurement of nursing care. Med Care Res Rev. 2007;64(2_suppl):10S–43S.
Dufour É, Duhoux A, Contandriopoulos D. Reliability of a Canadian database for primary care nursing services’ clinical and administrative data. Int J Med Inform. 2018;117:1–5.
Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, Finch T, Kennedy A, Mair F, O’Donnell C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8:63.
Sandelowski M, Voils CI, Barroso J. Defining and designing mixed research synthesis studies. Res Sch. 2006;13(1):29–40.
Pearson A, White H, Bath-Hextall F, Salmond S, Apostolo J, Kirkpatrick P. A mixed-methods approach to systematic reviews. Int J Evid-Based Healthc. 2015;13(3):121–31.
Sandelowski M, Leeman J, Knafl K, Crandell JL. Text-in-context: a method for extracting findings in mixed-methods mixed research synthesis studies. J Adv Nurs. 2013;69(6):1428–37.
Voils CI, Sandelowski M, Barroso J, Hasselblad V. Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field methods. 2008;20(1):3–25.
Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, Britten N, Roen K, Duffy S. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods program. Lancaster University: Lancaster Institute of Health Research; 2006.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.
Cooper H. Research synthesis and meta-analysis. A step-by-step approach. 5th ed. Thousand Oaks: Sage Publications; 2017.
Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35:29–45.
Pluye P, Robert E, Cargo M, Bartlett G, O’Cathain A, Griffiths F, Boardman F, Gagnon MP, Rousseau MC. Proposal: a mixed methods appraisal tool for systematic mixed studies reviews. Montreal: Department of Family Medicine, McGill University; 2011.
Joanna Briggs Institute. Joanna Briggs Institute Reviewers’ Manual: 2014 edition. Adelaide: The Joanna Briggs Institute; 2014.
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005;10(1):45–53.
Joanna Briggs Institute. New JBI grades of recommendation. Adelaide: The Joanna Briggs Institute; 2014.
Joanna Briggs Institute. New JBI levels of evidence. Adelaide: The Joanna Briggs Institute; 2014.
Joanna Briggs Institute. Supporting document for the Joanna Briggs Institute levels of evidence and grades of recommendation. Adelaide: The Joanna Briggs Institute; 2014.
We would like to thank Assia Mourid, librarian at the paramedical library of the University of Montreal, for the support in developing the search strategy.
This review did not receive any specific funding from financing of the public, commercial or non-profit sector.
Availability of data and materials
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Search strategy for CINAHL. (DOCX 24 kb)