- Open Access
Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation
Systematic Reviews volume 9, Article number: 21 (2020)
Stakeholder engagement has become widely accepted as a necessary component of guideline development and implementation. While frameworks for developing guidelines express the need for those potentially affected by guideline recommendations to be involved in their development, there is a lack of consensus on how this should be done in practice. Further, there is a lack of guidance on how to equitably and meaningfully engage multiple stakeholders. We aim to develop guidance for the meaningful and equitable engagement of multiple stakeholders in guideline development and implementation.
This will be a multi-stage project. The first stage is to conduct a series of four systematic reviews. These will (1) describe existing guidance and methods for stakeholder engagement in guideline development and implementation, (2) characterize barriers and facilitators to stakeholder engagement in guideline development and implementation, (3) explore the impact of stakeholder engagement on guideline development and implementation, and (4) identify issues related to conflicts of interest when engaging multiple stakeholders in guideline development and implementation.
We will collaborate with our multiple and diverse stakeholders to develop guidance for multi-stakeholder engagement in guideline development and implementation. We will use the results of the systematic reviews to develop a candidate list of draft guidance recommendations and will seek broad feedback on the draft guidance via an online survey of guideline developers and external stakeholders. An invited group of representatives from all stakeholder groups will discuss the results of the survey at a consensus meeting which will inform the development of the final guidance papers.
Our overall goal is to improve the development of guidelines through meaningful and equitable multi-stakeholder engagement, and subsequently to improve health outcomes and reduce inequities in health.
Existing guidance on stakeholder engagement largely focuses on patient, consumer, caregiver, or family involvement in guideline development but does not provide guidance on multi-stakeholder engagement.
The MuSE project will compile and provide evidence-based strategies for the equitable and meaningful engagement of multiple stakeholders during the guideline development and implementation process.
The MuSE project will develop a stakeholder engagement plan for its own project stakeholders and evaluate its development and implementation.
Guidelines from various entities exist for many health conditions and health-related activities. Guidelines contain recommendations for health practice, public health, or health policy . They are used by health care providers as well as policymakers, health system leaders, professional medical bodies, service organizations, funders, and regulatory authorities . Recommendations in guidelines should be based on available research evidence . Guideline development requires evaluating, summarizing, and making recommendations based on the available body of evidence regarding patient care, public health, and health systems. This requires weighing the benefits and risks that accompany all care and policy options before making recommendations . Sonnad  noted that a lack of connection between guideline developers and those who use them often leads to controversy and uncertainty. Indeed, engaging guideline users during the guideline development process has been noted to improve guideline recommendation uptake [5, 6]. Schünemann and colleagues , for one, consider implementation in policy and practice as part of the guideline development process (steps 14 and 16). Throughout this work, we will use guideline development to encompass the guideline development and implementation processes.
In recent years, we have witnessed considerable shifts in how healthcare research is planned, delivered, shared, and evaluated. It is now increasingly expected that individuals or groups involved in or affected by health- and healthcare-related decisions, programs, or policies (termed “stakeholders”) should have a say in the planning, conduct, dissemination, uptake, and evaluation of healthcare research. In other words, stakeholders should be engaged in the entire process of guideline development. Several entities [8,9,10,11], including the World Health Organization (WHO) and the National Institute for Health and Care Excellence (NICE), recommend involving stakeholders in guideline development. As such, stakeholder engagement has become widely accepted in the production of trustworthy guidelines [12, 13]. In guideline development, stakeholder engagement is considered critical to ensuring priority guideline topics are identified and that comprehensive assessments of the evidence and other considerations are done [14,15,16].
Stakeholder involvement can help to ensure a guideline’s acceptability and feasibility to the end users. They can also ensure that equity and human rights issues are taken into consideration and support the adoption of its recommendations into policy and practice. This in turn may lead to improved adherence to any treatments and practices recommended [7, 17]. Stakeholder engagement in guideline development is part of a wider acceptance by the research community of the value of ensuring the participation of end users in the research and knowledge translation cycles [18, 19]. There is a moral imperative to engage end users in that people have a right to be involved in activities that may affect them. End user engagement may also improve the relevance, transparency, and usefulness of guidelines .
There are many stakeholder groups equally affected by recommendations in guidelines—e.g., patients, consumers, providers, general public, researchers, and policymakers. However, engagement with patients/public/community stakeholder groups dominates the literature, and guidance of the engagement with patient/public stakeholders is the most prominent [21,22,23]. In a review of guideline methodologies conducted by Armstrong and Bloom for example, patients/public stakeholders were consulted by 101 different guideline developers . Many guideline groups that have sought to involve stakeholders have utilized limited numbers of participants or utilized slow and labor-intensive processes (e.g., time and resources needed to administer, collate, and respond to over 200 stakeholder views and comments) . It is recognized that successful guideline development and implementation requires the engagement of multiple stakeholders  and “shared solutions” (input from patients, clinicians, and policymakers) improve health outcomes [24,25,26]. Patient/public stakeholders may potentially feel intimidated to contribute if they are only one voice among many. Keeping patient and public stakeholder voices separate from other stakeholder groups potentially shortchanges the input and influence that this group may offer. Equitable engagement of multiple stakeholder groups can help to ensure that guidelines contribute to reducing health disparities [27, 28]. However, there is a lack of consensus on how to identify and recruit relevant stakeholders, how they should be engaged, what their roles and responsibilities should be, how to evaluate the impact of their engagement in guideline development, and how to best collect and manage conflicts of interest as part of the engagement and guideline development process.
Schünemann et al. identified 18 steps in the guideline development process, based on a review of 35 guideline manuals published between 2003 and 2013 . Several manuals mentioned the importance of including stakeholders, but few provided details on what stakeholder engagement should entail. In a review of 56 guidance documents for guideline development, 72% mentioned incorporating patients and their views in the process. However, the review did not provide sufficient detail on how to do this for each step of the guideline development process . Armstrong et al. developed a framework for continuous patient engagement in clinical practice guideline development that outlines options for patient engagement in the steps in which they are most commonly involved . While providing guidance on when to involve patients, the framework does not provide guidance on how to identify patients to participate and does not discuss other stakeholders.
Effective stakeholder engagement, ideally, facilitates the equitable contribution of relevant stakeholder groups in the guideline development and implementation process. This requires that guideline developers establish processes that prevent more financially powerful, highly vocal, or intellectually conflicted stakeholder groups from dominating the guideline development process. The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) equity guidelines, for one, recommend the inclusion of underrepresented stakeholder groups in the guideline development and implementation process . Stakeholders will have various levels of time, resources, and skills available to dedicate to the process, and ensuring that these differences do not result in certain stakeholder having more influence over the final guideline recommendations is important. A challenging aspect of engaging stakeholders is ensuring that interests are declared and that conflicts of interest are appropriately managed. Despite the importance of conflict of interest for guideline development, there is high variability in the process of disclosure and management of such conflicts across different organizations . As a consequence, there are many inconsistencies in how stakeholder engagement is considered across guideline development groups. There should be opportunities for a variety of opinions to be heard, but it is vital that the recommendations made are objective and not unduly influenced by vested interests. Strategies such as active outreach activities, giving adequate time for comment on guideline recommendations and using processes with follow-up prompts that ensure all stakeholder comments are systematically addressed may assist in reducing potential inequities and increase guideline development transparency .
The study aim is to develop guidance for guideline developers that supports the equitable and meaningful engagement of multiple stakeholders throughout the guideline development and implementation process. Guideline development will be used to encompass development, implementation, and evaluation processes. The objectives, in terms of participants, interventions, and comparators, are stakeholder/stakeholder groups engaged in guideline development, engagement in the guideline development process, and no comparator. Outcomes are outlined below for each phase of the project.
We define below the terms guideline, stakeholder, stakeholder engagement, levels of engagement, and under-represented groups used for this project.
Guidelines are “systematically developed evidence-based statements which assist providers, recipients, and other stakeholders informed decisions about appropriate health interventions” .
A stakeholder is any “individual or group who is responsible for or affected by health- and healthcare-related decisions that can be informed by research evidence” . Further, we acknowledge that some stakeholders may use guidelines to inform decision-making (CIHR terms these stakeholders, knowledge users), while others may have an interest in the recommendations for other reasons [32,33,34]. We define eight stakeholder groups for this project, namely (1) persons and the public (e.g., patients, their caregivers, families, and patient and consumer advocacy organizations), (2) providers (individuals/organizations that provide care, e.g., nurses, physicians, pharmacists, mental health counselors, community-based workers), (3) payers (pays for or reimburses for health-related interventions, e.g., insurers, individuals with deductibles, others responsible for reimbursement for health-related interventions), (4) purchasers (e.g., employers, self-insured, governments, and other entities responsible for underwriting the cost of care), (5) policymakers (policymaking entities such as governments and professional associations), (6) product makers (e.g., drug/device manufacturers), (7) principal investigators (e.g., researchers), and (8) the press (e.g., publishers, news media) [32, 33].
“Engagement” refers to the approach to gather input or contribution from stakeholders “toward the development of a guideline, completion of any stages of a guideline, or dissemination, uptake or evaluation of a guideline and its recommendations” . Engagement is considered multi-directional, resulting in “informed decision-making about the selection, conduct, and use of the research” . Depending on the context, engagement may also be termed collaboration, involvement, or partnership . Herein, we will use the term “stakeholder engagement.”
Levels of engagement (Table 1)
The extent to which stakeholders are engaged in the guideline development process can vary. We identify four levels of engagement (see Additional file 1), adapted from previous work [37,38,39]: (1) Communication—stakeholders receive information but have no role in contributing; (2) Consultation—stakeholders provide their views, thoughts, feedback, opinions, or experiences but without a commitment to act on them; (3) Collaboration—stakeholders are engaged to influence the production of the guidelines (e.g., commenting, advising, ranking, voting, prioritizing, and reaching consensus) but without direct control over decisions; and (4) Coproduction—stakeholders are equal members of the guideline development team and participate in all steps of the guideline development process. Members benefit from each other’s knowledge, skills, and perspectives and build relationships in an open, trusting, and transparent atmosphere that encourages learning from each other. With ongoing collaboration and engagement, all members have an equal opportunity to influence each aspect of the guideline development process [25, 35, 37, 40,41,42].
Under-represented groups refer to those individuals or groups who may experience health inequities for reasons such as a lack of inclusion in research, health policy, or guideline development; barriers to access of health services; or because of other socially stratifying factors, such as their place of residence, race/ethnicity/culture/language, occupation, gender/sex, subject matter knowledge, religion, education, socioeconomic status, social capital, age, or other individual characteristics [27, 28].
Study design and methods
We adapted our methods from the guidance for developing research reporting guidelines by Moher et al.  and our internal Terms of Reference for GRADE project groups document. Moher and colleagues recommend identifying the need for guidance, reviewing the literature, identifying participants, conducting a Delphi survey to gather opinions and set priorities, and holding a face-to-face consensus meeting. We have made a slight modification by adding key informant interviews to gather opinions on what should be included in the preliminary guidance. The conception and design of the study is the product of a global consortium for Multi-Stakeholder Engagement, entitled MuSE.
The MuSE consortium
The MuSE consortium was established in 2015 and includes over 80 researchers and stakeholders in various countries including: Australia, Brazil, Canada, Germany, Italy, Lebanon, the Netherlands, the Philippines, Switzerland, the UK, and the USA. The team includes researchers, policymakers, guideline developers, research funders, clinicians, patients and patient representatives, and policymakers from various organizations including the Agency for Healthcare Research and Quality (AHRQ), the Campbell Collaboration, Cochrane, GRADE Working Group, Health Canada, Patient-Centered Outcomes Research Institute (PCORI), Research and Development (RAND) Corporation, the World Health Organization (WHO), Joanna Briggs Institute (JBI), and multiple universities. All team members share an interest in developing methods and approaches for involving patients and other stakeholders in health outcomes research . The MuSE consortium includes three working groups with each undertaking projects related to the development of methods for involving multiple stakeholders in health outcomes research (see Additional file 2).
Stakeholder engagement in the MuSE project
We will develop a stakeholder engagement framework for this study that outlines how we will engage and evaluate our own stakeholder engagement processes throughout the project, drawing on principles of realist evaluation to explore what works, for whom, why, and in what context . We have adopted a co-production approach, drawing on the National Institute for Health Research (NIHR) guidance which pays attention to sharing power, including the perspectives and skills of all involved, respecting and valuing the knowledge of all, reciprocity, and building and maintaining relationships [25, 37, 40,41,42, 46]. We will work with our stakeholder to agree on the best ways to operationalize those key elements of co-production. We have assembled a large, international project team of co-investigators, collaborators, and stakeholders representing the various stakeholder groups as part of the MuSE consortium. All members of the consortium will be invited to provide advice and collaborate throughout the planning of each project stage, interpreting results, and coproducing the final guidance paper. To ensure accurate and transparent reporting of our stakeholder engagement throughout this project, we will follow the Guidance for Reporting Involvement of Patients and the Public (GRIPP2) checklist . We will document and report on the methods used to engage our stakeholders, the results of the stakeholder engagement, the extent to which stakeholders’ input influenced the guidance development process and outcomes, and the lessons learned from the experience .
GIN McMaster Guideline Development Checklist
This study will use the 18 steps contained in Schünemann et al.’s Guidelines 2.0 checklist, known as the GIN McMaster Guideline Development Checklist, for guideline development as its organizing framework . It provides guideline developers with a comprehensive checklist of items linked to relevant resources and tools to facilitate the guideline development process. The GIN McMaster Guideline Development Checklist has been used by guideline developers in various settings and was used to develop extensions for specific aspects of guideline development such as adaptation and rapid guideline development [48,49,50,51,52]. The 18 steps begin with the organization, budgeting, and planning of the development process, and continue with steps for priority setting, guideline group membership, question generation, all the way to developing recommendations, dissemination, implementation, and evaluation of the guidelines. Item six on the checklist relates to consumer and stakeholder involvement. As opposed to identifying stakeholder engagement as one step along the multi-step process of guideline development, the MuSE study will seek to document evidence-based guidance for stakeholder engagement for each step in the guideline development process (see Additional file 3).
Equity and the PROGRESS-Plus framework
The Canadian Institutes for Health Research (CIHR) expects that all research applicants will integrate gender and sex into their research designs when appropriate . The MuSE project recognizes gender as an important social determinant of health and contributor to health inequities. Gender refers to “the socially constructed characteristics of women and men—such as norms, roles, and relationships of and between groups of women and men. It varies from society to society and can be changed” . Sex is “the different biological and physiological characteristics of males and females, such as reproductive organs, chromosomes, hormones, etc.” . Gender considerations will be integrated throughout the MuSE project as part of the project’s adoption of the PROGRESS-Plus framework for equity. The PROGRESS-Plus framework identifies socially stratifying factors that can contribute to health inequities . We define health inequities as difference in health status that is avoidable, unfair, and unjust . PROGRESS refers to Place of residence, Race/ethnicity/culture/language, Occupation, Gender/sex, Religion, Education, Socioeconomic status, and Social capital. The Plus extends the original framework to include personal characteristics associated with discrimination (e.g., age, disability), features of relationships (e.g., children of smoking parents), and time-dependent relationships (e.g., release from incarceration) . We will use the framework to guide the integration of health equity considerations throughout the project. Below, we describe the project stages and the integration of equity in each.
This study includes various stages (Fig. 1). Briefly, they are (1) the conduct of four concurrent systematic reviews, (2) the development of draft guidance, (3) an online international survey of external stakeholders and experts, (4) a consensus meeting of project stakeholders, and (5) the finalization of the guidance paper. This protocol focuses on stage 1.
Stage 1: Systematic reviews of the literature
We will conduct four systematic reviews to identify and assess the available literature on stakeholder engagement in guideline development. This protocol will yield four distinct but related systematic reviews that will inform the overall protocol output. To improve efficiency, we will conduct an integrated literature search, screening, and data extraction process. Once included, articles will be tagged for their relevance to a particular review(s). Analyses and reports will be conducted separately for each review. Given the qualitative and multi-stage nature of this study, we believe the GRIPP2 checklist is more suitable for reporting the conduct of this project. The systematic review stage will be developed following the PRISMA-P checklist (Additional file 4)
Below, we briefly describe the objectives of each review and the methods common to all reviews.
Review 1: Existing guidance on stakeholder engagement in guideline development
The objective of this review is to synthesize existing guidance for stakeholder engagement at each of the 18 steps in the guideline development process. Eligible studies will describe the development of a framework or process for stakeholder engagement in guideline development. This includes frameworks or processes that address the implementation or evaluation of stakeholder engagement in guideline development. Outcomes to be synthesized are methods for (1) identifying stakeholders, (2) engaging stakeholders at different stages of the guideline development process, and (3) resolving differences in opinions/perspectives. In addition, frequency and level of engagement of stakeholders at each step of the guideline development process, level of engagement in each step of the guideline development process, and how stakeholders contributed at each step of the guideline development process will be synthesized.
Review 2: Barriers and facilitators to stakeholder engagement in guideline development
The expected result of this review is to summarize the barriers and facilitators to stakeholder engagement at each step in the guideline development process. Eligible studies will describe or assess the barriers and facilitators to stakeholder engagement in guideline development. A “barrier” is defined as any variable or condition that impedes stakeholder engagement in guideline development or implementation. A “facilitator” is defined as any variable or condition that promotes stakeholder engagement in guideline development or implementation.
Review 3: Conflicts of interest of multiple stakeholders in guideline development
The objective is to systematically review the literature on conflict of interest issues when engaging stakeholders in guideline development. Specifically, this review aims to answer the following questions: (1) What are the types of conflicts of interest that stakeholders engaged in the guideline development process have and how do they vary by stakeholder group? (2) Can the conflicts of interest of individuals or organizations selecting stakeholders to participate in the development of a guideline affect that selection process? (3) What are the potential effect(s) of conflicts of interest of stakeholders on the guideline development process? (4) What are the proposed and/or implemented approaches for managing the conflicts of interest of stakeholders engaged in guideline development, and what are their respective advantages and disadvantages?
Review 4: Impact of stakeholder engagement in guideline development
This review will assess the impact of stakeholder engagement in guideline development on (a) the guideline development process; (b) guideline relevance, trustworthiness, acceptability, and uptake; and (c) the stakeholders and panel members themselves. Eligible studies will identify or assess the impact of multi-stakeholder engagement through all the stages of guideline development.
Included studies will discuss stakeholder engagement in guideline development that assesses existing guidance and methods for stakeholder engagement at each stage of the guideline development and implementation process, barriers and facilitators to stakeholder engagement, conflicts of interest of stakeholders in the guideline development and implementation process, and/or impact of stakeholder engagement on the guideline development and implementation process.
For our purposes, stakeholders in guideline development are as described above under the “Key definitions” section.
Eligible studies will involve/engage stakeholders in some role(s) during the guideline development process. Engagement and guidelines are as described above under the “Key definitions” section.
Studies that do not involve stakeholders in the guideline development process will not be eligible for inclusion in this project.
All reviews will include quantitative, qualitative, and mixed-methods studies. As such, our methods will follow the Cochrane Handbook for Systematic Reviews of Interventions and the Handbook for Synthesizing Qualitative Research, as appropriate [57, 58]. We will include randomized trials, non-randomized studies (e.g., cohort studies, before and after studies, cross-sectional studies), qualitative studies, theoretical and ethical papers, process evaluation studies, policy analysis studies, case studies, and mixed-methods studies. We will exclude editorials, commentaries, proposals, and conference abstracts. Studies without a clear methods section will be excluded.
We will develop one comprehensive search strategy to identify relevant studies for the four reviews. We will search the following databases: MEDLINE (OVID), CINAHL (EBSCO), EMBASE (OVID), PsycInfo (OVID), AMED (OVID), and SCOPUS. We will not place limits on language, date, or study design. In addition, we will perform both backward and forward citation tracking to identify further eligible studies. A draft of the search strategy is provided (see Additional file 5).
To identify grey literature, we will search the websites of agencies who actively engage stakeholder groups such as the AHRQ, Canadian Institute of Health Research (CIHR) Strategy for Patient-Oriented Research (SPOR), INVOLVE, the National Institute for Health and Care Excellence (NICE), and the PCORI. We will also search the websites of guideline-producing agencies, such as the American Academy of Pediatrics, Australia’s National Health Medical Research Council (NHMRC), and the WHO. We will invite members of the team to suggest grey literature sources, and we plan to broaden the search by soliciting suggestions via social media, such as Twitter (https://twitter.com/CochraneEquity).
Titles and abstracts of the studies identified by the search strategy and the full texts of those assessed as potentially relevant will be screened independently, in duplicate using Covidence software . Disagreements on study selection will be resolved by discussion or with a third member of the research team when necessary. Eligible studies will be exported into an Excel spreadsheet and independently “tagged” for their relevance to each review. One study may be included in more than one review. For example, a study may describe a process for stakeholder engagement in guideline development as well as describe the barriers and facilitators to the stakeholder engagement.
The data extraction form will be pre-tested and will include (as applicable) factors related to the population, intervention, comparison, and outcomes. Extracted data common to all reviews include information on study characteristics (e.g., year of publication, authors, type of publication), stakeholders included, stakeholder roles, and stakeholder characteristics. The data will be extracted independently in duplicate by two reviewers and will be piloted on ten articles. Disagreements on extractions will be resolved by discussion and with a third member of the research team when necessary. Where necessary, the corresponding author of eligible studies will be contacted for additional information
We will use the 18 steps of the guideline development process as a framework for data extraction and will map the existing guidance, barriers and facilitators, conflicts of interest, and impacts discussed in our included studies to one or more of the steps of this framework . Further, we will extract data that describe study populations and results or findings by PROGRESS-Plus factors.
Risk of bias
We will examine the methodological quality of the included studies as appropriate. We will use the risk of bias tools from the Cochrane Handbook for randomized trials (ROB2) , the Risk of Bias in Non-randomized Studies – of Interventions (ROBINS-I) tool , and the Critical Appraisal Skills Programme (CASP) qualitative appraisal research tool  for qualitative studies. Risk of bias will be assessed independently, in duplicate, by two authors, and any discrepancies will be resolved by consensus and consultation with a third author, when necessary.
We will use a mixed-methods approach to summarize our findings. Qualitative and quantitative studies will be analyzed and synthesized separately, while the implications for practice, policy, and research that will form the discussion and conclusion sections of the reviews will draw on both the qualitative and quantitative syntheses. We will summarize findings across the 18-steps of the GIN-McMaster Checklist, the eight stakeholder groups, and the PROGRESS-Plus factors. We will report the systematic reviews following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA)  and the Enhancing Transparency in Reporting the Synthesis of Qualitative Research (ENTREQ)  reporting guidelines, as appropriate. The GRADE or Confidence in the Evidence from Reviews of Qualitative Research (CERQual) methodology will be used to evaluate the quality of evidence for each review as appropriate.
Stage 2: Drafting the guidance
In consultation with the MuSE consortium, we will use the results of the four systematic reviews to develop a candidate list of recommendations to include in the final guidance paper. These draft recommendations will provide guidance mapped to the 18 steps of the guideline development process which will allow guideline developers to consider when and how to engage different stakeholders at each stage of the guideline development process. Equity considerations at each step of the guideline development process will be outlined.
Drafting and refining the guidance will involve structured engagement similar to that described by Jull and colleagues  with all MuSE consortium members. Because of our large and geographically diverse team, we will consult with members via email, teleconferences, and face-to-face meetings, as appropriate, to revise the guidance and ensure that all team members have equal opportunity to contribute and influence the research output. We will work with our MuSE consortium members to equitably include diverse voices in the drafting process, including visible minorities, women, members from low- and middle-income countries, and others who are traditionally omitted from guideline development processes.
Stakeholder engagement should be multidirectional, meaningful, effective, and enable equity for both the stakeholders and guideline developers throughout all steps in the decision-making process. This project is innovative in that we are committed to an inclusive, comprehensive, and equitable approach to ensure that the guidance we develop is representative and relevant for all stakeholders, including those who are involved in creating and implementing guidelines and those affected by the recommendations developed within those guidelines. Our stakeholder engagement strategy for the project will assist us in effectively engaging our own stakeholders and allow us to monitor our engagement processes in real-time so that course corrections can be made if there is evidence of non-meaningful engagement.
We are not including language or date restrictions on our search strategy. A potential limitation of these reviews is that we are including papers regardless of their methodological quality. This will allow us to collect and synthesize qualitative data that we may miss if standard methodological criteria are applied. We will discuss limitations further in the full review.
These reviews will contribute to the literature by identifying existing guidance, barriers and facilitators, potential impacts, and possible conflict of interest issues related to engagement of stakeholders from many stakeholder groups in guideline development and implementation.
The results of the four reviews will inform the development of draft guidance. Once drafted, we will gather opinions and priorities on the guidance items from a wide range of purposefully selected stakeholders external to the MuSE consortium, including representation from low- and middle-income countries through an anonymous, online survey. We will strive for equity in the identification of survey recipients by engaging with a diversity of respondents representing different physical capabilities, genders, geographies, socio-economic statuses, and ethnicities. We will then present the results of the survey for each candidate item and use structured discussions to reach consensus on the included items for the final guidance paper at a two-day face-to-face consensus meeting as recommended by Moher et al. . Finally, based on the results of the previous stages, we will develop guidance that provides recommendations for stakeholder roles and modes of engagement at different steps of the guideline development process (including implementation and evaluation) and for managing conflicts of interest. We will use an iterative process of feedback to draft, refine, and finalize the guidance to be provided in each manuscript in consultation with the co-authors of each paper and the other members of the MuSE consortium. The final product will be included in GRADE Working Group Guidance. The GRADE Working Group has developed internationally recognized guidance for the development of clinical practice and public health guidelines .
The expected final guidance will contribute to improving the guideline development and implementation process by identifying strategies for the meaningful and equitable engagement of all relevant stakeholder groups at all stages. Through this project, we aim to contribute to the growing body of literature on stakeholder engagement for better quality guidance, increased uptake of guidance, more relevant health programs, policies and services, and more equitable health outcomes.
We continually welcome additional expressions of interest and suggestions for relevant literature and plan to evaluate our own stakeholder engagement throughout this work to ensure meaningful engagement.
Availability of data and materials
Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.
Agency for Healthcare Research and Quality
Critical Appraisal Skills Programme
Confidence in the Evidence from Reviews of Qualitative Research
Canadian Institute of Health Research
Enhancing Transparency in Reporting the Synthesis of Qualitative Research
Grading of Recommendations, Assessment, Development and Evaluation
Guidance for Reporting Involvement of Patients and the Public
Joanna Briggs Institute
National Institute for Health and Care Excellence
National Institute for Health Research
Patient-Centered Outcomes Research Institute
Preferred Reporting Items for Systematic Reviews and Meta-Analysis
Place of residence, Race/ethnicity/culture/language, Occupation, Gender/sex, Religion, Education, Socioeconomic status, and Social capital
Risk of bias
Risk of Bias in Non-randomized Studies – of Interventions
Strategy for Patient-Oriented Research
World Health Organization
WHO. WHO Handbook for Guideline Development – 2nd Edition [Internet]. World Health Organization, Geneva, Switzerland; 2014 [cited 2019 Apr 5]. Available from: http://apps.who.int/medicinedocs/en/m/abstract/Js22083en/
Kredo T, Bernhardsson S, Machingaidze S, Young T, Louw Q, Ochodo E, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care J Int Soc Qual Health Care. 2016 Feb;28(1):122–8.
Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. Clinical practice guidelines we can trust [Internet]. Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, editors. Washington (DC): National Academies Press (US); 2011 [cited 2019 Apr 5]. Available from: http://www.ncbi.nlm.nih.gov/books/NBK209539/
Sonnad SS. Organizational tactics for the successful assimilation of medical practice guidelines. Health Care Manage Rev. 1998;23(3):30–7.
Moulding NT, Silagy CA, Weller DP. A framework for effective management of change in clinical practice: dissemination and implementation of clinical practice guidelines. BMJ Qual Saf. 1999 Sep 1;8(3):177–83.
Gagliardi AR, Brouwers MC. Integrating guideline development and implementation: analysis of guideline development manual instructions for generating implementation advice. Implement Sci IS. 2012 Jul 23;7:67.
Schünemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ Can Med Assoc J J Assoc Medicale Can. 2014;186(3):E123–42.
Jarret L, Patient Involvement Unit. A report on a study to evaluate patient/carer membership of the first NICE Guideline Development Groups [Internet]. National Institute for Clinical Excellence; 2004 [cited 2019 Apr 24]. Available from: https://www.nice.org.uk/media/default/About/NICE-Communities/Public-involvement/Public-involvement-programme/PIU-GDG-evaluation-report-2004-1.pdf
Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P, et al. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012 Apr 3;156(7):525–31.
Schünemann HJ, Fretheim A, Oxman AD, WHO. Advisory Committee on Health Research. Improving the use of research evidence in guideline development: 1. Guidelines for guidelines. Health Res Policy Syst. 2006 Nov 21;4:13.
Guidelines I of M (US) C on S for DTCP, Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E. Committee on standards for developing trustworthy clinical practice guidelines [Internet]. National Academies Press (US); 2011 [cited 2019 Apr 24]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK209544/
Cluzeau F, Wedzicha JA, Kelson M, Corn J, Kunz R, Walsh J, et al. Stakeholder involvement: how to do it right: article 9 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012 Dec;9(5):269–73.
NHMRC. 2016 NHMRC Standards for Guidelines | NHMRC [Internet]. 2016 [cited 2019 Apr 5]. Available from: https://www.nhmrc.gov.au/about-us/publications/2016-nhmrc-standards-guidelines
Gillard S, Simons L, Turner K, Lucock M, Edwards C. Patient and public involvement in the coproduction of knowledge: reflection on the analysis of qualitative data in a mental health study. Qual Health Res. 2012 Aug;22(8):1126–37.
Oliver K, Lorenc T, Innvær S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12:34.
Armstrong MJ, Rueda J-D, Gronseth GS, Mullins CD. Framework for enhancing clinical practice guidelines through continuous patient engagement. Health Expect Int J Public Particip Health Care Health Policy. 2017;20(1):3–10.
Carroll C. Qualitative evidence synthesis to improve implementation of clinical guidelines. BMJ. 2017 Jan 16;356:j80.
Jackson CL, Greenhalgh T. Co-creation: a new approach to optimising research impact? Med J Aust. 2015 Oct 5;203(7):283–4.
Ghaffar A, Langlois EV, Rasanathan K, Peterson S, Adedokun L, Tran NT. Strengthening health systems through embedded research. Bull World Health Organ. 2017 01;95(2):87.
Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015 Mar;4(2):133–45.
Armstrong MJ, Bloom JA. Patient involvement in guidelines is poor five years after institute of medicine standards: review of guideline methodologies. Res Involv Engagem. 2017;3(1):19.
Lavis JN, Paulsen EJ, Oxman AD, Moynihan R. Evidence-informed health policy 2 – Survey of organizations that support the use of research evidence. Implement Sci. 2008;3(1):54.
van de Bovenkamp HM, Zuiderent-Jerak T. An empirical study of patient participation in guideline development: exploring the potential for articulating patient knowledge in evidence-based epistemic settings. Health Expect. 2015;18(5):942–55.
Suman A, Dikkers MF, Schaafsma FG, van Tulder MW, Anema JR. Effectiveness of multifaceted implementation strategies for the implementation of back and neck pain guidelines in health care: a systematic review. Implement Sci. 2016 Sep 20;11(1):126.
Dunston R, Lee A, Boud D, Brodie P, Chiarella M. Co-production and health system reform - from re-imagining to re-making. Aust J Public Adm. 2009;68(1):39–52.
Kumarasamy MA, Sanfilippo FP. Breaking down silos: engaging students to help fix the US health care system. J Multidiscip Healthc. 2015;8:101–8.
Shi C, Tian J, Wang Q, Petkovic J, Ren D, Yang K, et al. How equity is addressed in clinical practice guidelines: a content analysis. BMJ Open. 2014;4(12):e005660.
Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010 Apr 1;100 Suppl 1:S40-S46.
Selva A, Sanabria AJ, Pequeño S, Zhang Y, Solà I, Pardo-Hernandez H, et al. Incorporating patients’ views in guideline development: a systematic review of guidance documents. J Clin Epidemiol. 2017;88:102–12.
Akl EA, Welch V, Pottie K, Eslava-Schmalbach J, Darzi A, Sola I, et al. GRADE equity guidelines 2: considering health equity in GRADE guideline development: equity extension of the guideline development checklist. J Clin Epidemiol. 2017;90:68–75.
Global Programme on Evidence for Health Policy. Guideline for WHO Guidelines [Internet]. World Health Organization, Geneva, Switzerland; 2003 [cited 2019 Apr 24]. Available from: http://archives.who.int/eml/expcom/expcom14/1other/guid_for_guid.pdf
Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise J-M, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012;27(8):985–91.
Tugwell P, Robinson V, Grimshaw J, Santesso N. Systematic reviews and knowledge translation. Bull World Health Organ. 2006;84(8):643–51.
Canadian Institutes of Health Research. Guide to knowledge translation planning at CIHR: integrated and end-of-grant approaches [Internet]. Ottawa, Ontario; 2012 [cited 2019 May 29]. Available from: http://www.cihr-irsc.gc.ca/e/documents/kt_lm_ktplan-en.pdf, http://www.cihr-irsc.gc.ca/e/documents/kt_lm_ktplan-en.pdf
Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, et al. Stakeholder involvement in systematic reviews: a scoping review. Syst Rev. 2018;7(1):208.
Hoddinott P, Pollock A, O’Cathain A, Boyer I, Taylor J, MacDonald C, et al. How to incorporate patient and public perspectives into the design and conduct of research. F1000Research. 2018 Jun 18;7:752.
Oliver SR, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect Int J Public Particip Health Care Health Policy. 2008;11(1):72–84.
Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, et al. Development of the ACTIVE framework to describe stakeholder involvement in systematic reviews. J Health Serv Res Policy. 2019;18:1355819619841647.
Crowe S. “Who inspired my thinking?- Sherry Arnstein.” Res All. 2017;1(1):143-146(4).
INVOLVE-National Institute for Health Research. Guidance on co-producing a research project [Internet]. Hampshire, UK; 2018 [cited 2019 Jun 7]. Available from: https://www.invo.org.uk/posttypepublication/guidance-on-co-producing-a-research-project/
Markkanen S, Burgess G. Introduction to co-production in research: summary report; 2016.
Pohl C, Rist S, Zimmermann A, Fry P, Gurung GS, Schneider F, et al. Researchers’ roles in knowledge co-production: experience from sustainability research in Kenya, Switzerland. Bolivia and Nepal. Sci Public Policy. 2010;37(4):267–81.
Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.
Concannon TW, Grant S, Welch V, Petkovic J, Selby J, Crowe S, et al. Practical guidance for involving stakeholders in health research. J Gen Intern Med. 2019;34(3):458–63.
Wilson P, Mathie E, Keenan J, McNeilly E, Goodman C, Howe A, et al. ReseArch with Patient and Public invOlvement: a RealisT evaluation – the RAPPORT study [Internet]. Southampton (UK): NIHR Journals Library; 2015 [cited 2019 Jun 12]. (Health Services and Delivery Research). Available from: http://www.ncbi.nlm.nih.gov/books/NBK315999/
National Institute for Health Research. National Standards for Public Involvement In Research [Internet]. 2018 [cited 2019 Jun 12]. Available from: https://www.nihr.ac.uk/news-and-events/documents/Public_Involvement_Standards_March%202018_WEB.pdf
Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. Res Involv Engagem. 2017;3:13.
Falzon D, Schünemann HJ, Harausz E, González-Angulo L, Lienhardt C, Jaramillo E, et al. World Health Organization treatment guidelines for drug-resistant tuberculosis, 2016 update. Eur Respir J. 2017;49(3).
Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016 Jun 30;353:i2089.
Fiocchi A, Pawankar R, Cuello-Garcia C, Ahn K, Al-Hammadi S, Agarwal A, et al. World Allergy Organization-McMaster University guidelines for allergic disease prevention (GLAD-P): probiotics. World Allergy Organ J. 2015;8(1):4.
Laine L, Kaltenbach T, Barkun A, McQuaid KR, Subramanian V, Soetikno R, et al. SCENIC international consensus statement on surveillance and management of dysplasia in inflammatory bowel disease. Gastrointest Endosc. 2015 Mar;81(3):489-501.e26.
Farrell B, Tsang C, Raman-Wilms L, Irving H, Conklin J, Pottie K. What are priorities for deprescribing for elderly patients? Capturing the voice of practitioners: a modified delphi process. PloS One. 2015;10(4):e0122246.
Canadian Institutes of Health Research. How to integrate sex and gender into research [Internet]. 2018 [cited 2019 Jun 11]. Available from: http://www.cihr-irsc.gc.ca/e/50836.html
World Health Organization. Gender mainstreaming for health managers: a practical approach [Internet]. Geneva : World Health Organisation; 2011 [cited 2019 Jun 11]. Available from: https://apps.who.int/iris/handle/10665/44516
O’Neill J, Tabish H, Welch V, Petticrew M, Pottie K, Clarke M, et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol. 2014 Jan 1;67(1):56–64.
Whitehead M. The concepts and principles of equity and health. Int J Health Serv. 1992;22(3):429–45.
Sandelowski M, Barroso J. Handbook for synthesizing qualitative research. New York: Springer Publishing Company; 2007. 284 p.
Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions [Internet]. The Cochrane Collaboration; 2011 [cited 2019 Apr 5]. Available from: https://handbook-5-1.cochrane.org/
Higgins JPT, Sterne JAC, Savović j., Page MJ, Hróbjartsson A, Boutron I, et al. Appraising the risk of bias in randomized trials using the Cochrane Risk of Bias Tool. In: Cochrane Methods 2016 J Chandler, J McKenzie, I Boutron and V Welch (editors) Cochrane Database of Systematic Reviews 10 (Suppl 1) [Internet]. Available from: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD201601/full
Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.
Critical Appraisal Skills Programme. Casp Checklists. Casp. 2013.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.
Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12:181.
Jull J, Petticrew M, Kristjansson E, Yoganathan M, Petkovic J, Tugwell P, et al. Engaging knowledge users in development of the CONSORT-Equity 2017 reporting guideline: a qualitative study using in-depth interviews. Res Involv Engagem. 2018;4:34.
Hultcrantz M, Rind D, Akl EA, Treweek S, Mustafa RA, Iorio A, et al. The GRADE Working Group clarifies the construct of certainty of evidence. J Clin Epidemiol. 2017;87:4–13.
We would like to thank the entire MuSE consortium membership for their valuable contributions and advice.
This work has been funded by the Canadian Institutes of Health Research (PJT-155970; Application number: 391110). AS is supported by a National Health and Medical Research Council Postgraduate Scholarship (GNT 1132803). SVK is funded by a NRS Senior Clinical Fellowship (SCAF/15/02), the Medical Research Council (MRC_UU_12017/13 & MRC_UU_12017/15), and the Scottish Government Chief Scientist Office (SPHSU13 & SPHSU15). PC and AP are employed by the Nursing, Midwifery and Allied Health Professions Research Unit, which is funded by the Chief Scientist Office in Scotland. IDG is a recipient of a Canadian Institutes of Health Research Foundation Grant (FDN# 143237). The funders had no role in study design, preparation of manuscript, or the decision to publish.
Ethics approval and consent to participate
The international online survey stage of the study will be submitted to the Bruyère Research Institute’s Ethics Review Board for approval prior to launching the survey.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1: Table S1.
Levels of engagement.
Additional file 2.
The MuSE Consortium.
Additional file 3.
PRISMA-P reporting checklist.
Additional file 4.
Stakeholders across the 18 steps.
Additional file 5.
Draft Search Strategy in Medline.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Petkovic, J., Riddle, A., Akl, E.A. et al. Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation. Syst Rev 9, 21 (2020). https://doi.org/10.1186/s13643-020-1272-5
- Stakeholder engagement
- Systematic reviews
- Integrated knowledge translation