Skip to main content

Strategies for involving patients and the public in scaling-up initiatives in health and social services: protocol for a scoping review and Delphi survey

Abstract

Background

The scale-up of evidence-based innovations is required to reduce waste and inequities in health and social services (HSS). However, it often tends to be a top-down process initiated by policy makers, and the values of the intended beneficiaries are forgotten. Involving multiple stakeholders including patients and the public in the scaling-up process is thus essential but highly complex. We propose to identify relevant strategies for meaningfully and equitably involving patients and the public in the science and practice of scaling up in HSS.

Methods

We will adapt our overall method from the RAND/UCLA Appropriateness Method. Following this, we will perform a two-prong study design (knowledge synthesis and Delphi study) grounded in an integrated knowledge translation approach. This approach involves extensive participation of a network of stakeholders interested in patient and public involvement (PPI) in scaling up and a multidisciplinary steering committee. We will conduct a systematic scoping review following the methodology recommended in the Joanna Briggs Institute Reviewers Manual. We will use the following eligibility criteria: (1) participants—any stakeholder involved in creating or testing a strategy for PPI; (2) intervention—any PPI strategy proposed for scaling-up initiatives; (3) comparator—no restriction; (4) outcomes: any process or outcome metrics related to PPI; and (5) setting—HSS. We will search electronic databases (e.g., Medline, Web of Science, Sociological Abstract) from inception onwards, hand search relevant websites, screen the reference lists of included records, and consult experts in the field. Two reviewers will independently select and extract eligible studies. We will summarize data quantitatively and qualitatively and report results using the PRISMA extension for Scoping Reviews (PRISMA-ScR) checklist. We will conduct an online Delphi survey to achieve consensus on the relevant strategies for PPI in scaling-up initiatives in HSS. Participants will include stakeholders from low-, middle-, and high-income countries. We anticipate that three rounds will allow an acceptable degree of agreement on research priorities.

Discussion

Our findings will advance understanding of how to meaningfully and equitably involve patients and the public in scaling-up initiatives for sustainable HSS.

Systematic review registration

We registered this protocol with the Open Science Framework on August 19, 2020 (https://osf.io/zqpx7/).

Peer Review reports

Background

There is widespread enthusiasm for the use of ideas, practices, or products perceived as new (i.e., innovations) and for which an evidence base has been established (i.e., evidence-based innovations) in order to improve health and social services (HSS) [1, 2]. The impact of evidence-based innovations (EBIs) should be optimized to reduce waste and inequities in HSS and improve the health of populations [3, 4]. But evidence is limited on how to implement EBIs so that they benefit more patients or a broader spectrum of patients in healthcare systems [5]. For example, there is a shortage of evidence on strategies for expanding or “scaling up” EBIs [6]. In knowledge translation (KT) and implementation science (both hereafter referred to as KT), the differences between “scaling up,” “scaling out,” “scaling deep,” “scaling,” and “spread” are variously nuanced [5,6,7]. Here, we use the term “scaling up” or “scale up,” which is more familiar to lay knowledge users (e.g., patients and the public). The World Health Organization defines the process of “scaling up” as “deliberate efforts to increase the impact of successfully tested health innovations so as to benefit more people and to foster policy and program development on a lasting basis” [8]. Here, we understand scaling up to optimize equity, for example, as enlarging the scope of EBI not only numerically but also in other respects, e.g., increasing its range to a wider variety of socio-economic backgrounds [5]. While anticipating scaling up should begin at the earliest stages of research [5, 9], we summarized the main scaling-up steps as follows [1]: (1) scalability assessment, (2) development of a scaling-up strategy, (3) implementation and evaluation of the strategy, and (4) promoting the long-term sustained use of the successfully scaled-up EBI. The scalability assessment, the preliminary and essential step in scaling up an EBI, refers to assessment of the “ability of a health innovation shown to be efficacious on a small scale and/or under controlled conditions to be expanded under real world conditions to reach a greater proportion [or range] of the eligible population, while retaining effectiveness” [5, 10].

Just as the scaling up of EBIs remains an understudied phase of KT [11, 12], the science and practice of scaling up, in turn, has not yet fully integrated patient-oriented research principles [6, 7]. Our previous knowledge synthesis shows that barriers to scaling up EBIs include an absence of patient and public involvement (PPI) [6]. Scaling up often tends to be a top-down process initiated by policy makers and/or researchers [6, 7]. The values, preferences, and perspectives of the intended beneficiaries, i.e., the patients and public, are often forgotten. As patients and the public are at the core of healthcare systems that serve them, it is essential that they be part of the scaling up of such systems [13, 14], including preparatory, execution, and translational phases [15]. Not only should they be involved, but their involvement should be meaningful and equitable, i.e., it should be more than tokenistic, and their input should carry equal weight as that of policy makers and researchers. Strengthening patient and public influence on health decisions at every level is a necessary condition for a continuously learning health care system [16, 17]. However, policy makers and researchers who want to involve lay participants in this domain dominated by experts face a multiple of challenges, which need to be addressed.

In this research, we aim to address this knowledge gap in KT by building patient-oriented research capacity in the science and practice of scaling up in HSS, by ensuring that patients and the public are meaningfully and equitably involved. Therefore, our research question is: “How can patients and the public be meaningfully and equitably involved in scaling-up initiatives in HSS?” Our specific objectives are to (1) review existing literature on PPI in any scaling-up initiative in HSS; (2) achieve consensus on relevant strategies for involving patients and the public at each scaling-up step in HSS; and (3) build an international network of interest in the involvement of patients, the public, and other stakeholders in the science and practice of scaling up in HSS. This effort will contribute to an evidence-driven and people-centered science of scaling up.

Methods

Overarching methodological approach

We will adapt our overall method from the RAND/UCLA Appropriateness Method [18, 19]. Following this, our research will entail a two-prong study design: (1) a knowledge synthesis and (2) a Delphi study. We will use the Canadian Institutes of Health Research (CIHR) guidance to integrate considerations of sex and gender [20] into both the knowledge synthesis and the Delphi study. Our project is grounded in an integrated KT (iKT) approach and thus involves extensive participation of (1) a network of stakeholders interested in PPI in scaling up, the Research on Patient-Oriented Scaling-up (RePOS) Network, and (2) a multidisciplinary steering committee. The latter will include at least two patients or citizens, one policy- or decision-maker, one health care provider, one researcher, one scientific coordinator, and one trainee. We will use the CIHR Model of Knowledge Translation [21], the Montreal model [14], and the Public and Patient Engagement Evaluation Tool (PPEET) [22] to guide and assess our iKT approach. Finally, to ensure accurate and transparent reporting of this iKT approach, we will report evaluations using the Guidance for Reporting Involvement of Patients and the Public (GRIPP2) checklist [23] and the Sex and Gender Equity in Research (SAGER) guidelines [24].

Knowledge synthesis

Study design

We will conduct a scoping review to identify strategies that have been proposed for involving patients and the public in the science or practice of scaling up innovations in HSS. Scoping reviews are defined as a type of knowledge synthesis that follows a systematic approach to map evidence on a topic and identify main concepts, theories, sources, and knowledge gaps [25]. We will follow the methodology recommended in the Joanna Briggs Institute (JBI) Reviewers’ Manual for scoping reviews [26, 27] and have registered the protocol in the Open Science Framework (OSF) on August 19, 2020 (registration identifier: https://osf.io/zqpx7/) [28]. We report the content for this scoping review protocol using the Preferred Reporting Items for Systematic reviews and Meta-Analyses Protocol (PRISMA-P) checklist (Additional file 1) [29].

Eligibility criteria

Following the “PICOS” (participants, intervention, comparator, outcome, and setting) framework [30], we will use the following inclusion criteria:

  • ▪ Participants (P): Any type of stakeholder involved in creating or testing a strategy for PPI in the science or practice of scaling up. Stakeholder refers to “individual or group who is responsible for or affected by health- and healthcare-related decisions that can be informed by research evidence” [31, 32]. We will include the following nine types of stakeholders: (1) patients and the public (e.g., patients, their informal caregivers, their families, patient and consumer advocacy organizations, and local, national, or global citizens), (2) providers (e.g., individuals and organizations that provide care to patients and populations such as nurses, physicians, pharmacists, mental health counselors, and community-based workers), (3) purchasers (e.g., employers, governments, and other individuals or entities responsible for underwriting the cost of care), (4) payers (e.g., payers who pay or reimburse costs of health-related interventions such as insurers, individuals with deductibles, and others responsible for reimbursement for health-related\ interventions), (5) policy makers (e.g., policymaking entities such as governments and professional associations), (6) product makers (e.g., drug and device manufacturers), (7) investigators (e.g., researchers, postdoctoral fellows, and research coordinators), (8) the press (e.g., publishers and news media), and (9) trainees (e.g., university or college students) [31,32,33,34]. This includes organizations that provide guidance for promoting PPI in the science and practice of scaling up in HSS.

  • ▪ Intervention (I): Any strategy proposed for involving patients and/or the public (alone) or multiple stakeholders (including patients and/or the public) in research or practical initiatives related to the steps of the scaling-up process in HSS. We will consider the four levels of PPI [13, 14]: (1) information (i.e., patient or the public receives information but has no role in contributing); (2) consultation (i.e., patients or the public provide their views, thoughts, feedback, opinions, or experiences but without a commitment to act on them; (3) collaboration (i.e., patients or the public are engaged to influence the scaling-up initiative, including commenting, advising, ranking, voting, prioritizing, and reaching consensus, but without direct control over decisions; and (4) coproduction (i.e., patients or the public are equal members of the research team and participate in all steps of the scaling-up initiative).

  • ▪ Comparator (C): We will consider for inclusion both studies with comparison group and studies without.

  • ▪ Outcomes (O): We will consider a wide range of metrics related to PPI. We will use the following taxonomy: process metrics and outcome metrics [35]. The process metrics form four domains: (1) direct process metrics (e.g., agenda setting and time allocation, roles in decision making, and control over the meeting minutes); (2) surrogate process metrics (e.g., formal power, organizational commitment to involvement, and equality of participation); (3) preconditions for involvement metrics (e.g., literacy of patient and public participants, resources provided, and presence or quality of training); and (4) aggregate process metrics (e.g., respect, transparency of the decision-making process, and level or ladder of participation). The outcome metrics form three domains: (1) internal outcomes as measured by impact on involved patient or public participants (e.g., knowledge, satisfaction, and trust), on services provided (e.g., efficiency and cost-effectiveness of services, number of complaints about services, and services quality and safety), and on organization or system (e.g., funding and resources availability, staff views on PPI, and presence of racism in system); (2) external outcomes as measured by influence on the broader public (e.g., awareness or knowledge of health issues, support of the organization or system, and PPI as part of social change outside the organization), and on population health (e.g., population health status and level of health inequalities); and (3) aggregate outcomes (e.g., overall cost-effectiveness of PPI).

  • ▪ Setting (S): Any context related to HSS. The World Health Organization defines health as “a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity” [36].

Literature search

We will perform a comprehensive search for both published peer-reviewed records and “grey literature” through searching electronic databases, hand searching relevant websites, screening the reference lists of included records, and consulting experts in the field. First, we will search Medline, EMBASE, Web of Science, CINAHL, PsycINFO, ERIC, the Cochrane Library, Sociological Abstract, and Academic Search Premier from their inception onwards. Our information specialist will perform the electronic search strategy in Medline. The search terms will reflect the concepts: (1) PPI, (2) scaling up, and (3) HSS. These terms will be adapted to the above-mentioned databases. There will be no restriction with regard to language, date of publication, or type of document. This strategy will be reviewed by our international authors, experts on scaling up, and stakeholder involvement, then by a second information specialist using the Peer Review of Electronic Search Strategies (PRESS) guideline [37]. A draft of the search strategy can be found in Additional file 2. Second, we will consult the “grey literature” involving general and targeted Internet searches. This approach is promoted by the Agency for Healthcare Research and Quality (AHRQ) as a way of reducing publication bias [38]. Grey literature consists of unpublished literature, including publicly available information produced by all levels of government, academic institutions, business, and industry, in print and electronic formats, which is not controlled by commercial publishers [39,40,41]. For the general search, we will enter various configurations of a search term list into Google. For the targeted search, we will examine websites of the national and international organizations recommended by our steering committee and by other international scaling-up experts. Additionally, we will use the approach recommended by the Canadian Agency for Drugs and Technologies in Health (CADTH) for systematic searches of grey literature [42]. Finally, we will search published bibliographies of related topics and citations in included articles and consult experts in the field to identify additional relevant studies.

Study selection

We will operationalize our inclusion criteria using questions based on our PICOS elements. Two reviewers will perform a calibration exercise to ensure the criteria capture relevant studies. They will also independently screen titles and abstracts of a random sample of 5% of studies identified with our literature search. We will discuss the results of this pilot and review the eligibility criteria. We will check inter-reviewer agreement between these reviewers using the kappa statistic. Then, two reviewers will independently screen all remaining titles and abstracts for relevance, procure full texts of relevant studies, and select eligible studies. We will resolve any disagreements through consensus between the two reviewers and, if needed, with a third party. We will use the Covidence software for the selection of titles and abstracts [43]. Eligible studies will be exported into an Excel spreadsheet.

Data extraction

First, we will develop a data-charting form to guide extraction of variables based on the Montreal model [14], which is an adapted version of the framework for patient and family engagement in health [13]. The data-charting form will include the PICOS elements and “PROGRESS” (place of residence, race/ethnicity/culture/language, occupation, gender/sex, religion, education, socioeconomic status, and social capital) factors [44]. Second, two reviewers will perform a calibration exercise to guide selection of evidence sources to ensure the form captures all relevant data. Then, they will independently extract data from some included studies using the data-charting form and meet the steering committee members to determine whether the approach to data extraction is consistent with the research question and purpose. Third, we will separately extract strategies used for involving (1) patients and/or the public (alone) and (2) multiple stakeholders including patients and/or the public. We will also separately extract strategies used for women (e.g., for scaling up a prenatal screening program) from those used for men (e.g., for scaling up a prostate cancer program), and other sex- or gender-specific strategies. Finally, we will resolve any disagreements through consensus between the two reviewers and, if needed, with a third party. We will contact authors of included studies to provide missing or additional data if necessary.

Risk of bias

We will not perform an appraisal for risk of bias. Due to the nature of our research question, assessing risk of bias is not mandatory, according to the Joanna Briggs Institute Reviewers’ Manual [27, 45].

Data analysis

We will summarize data quantitatively (using frequencies) and qualitatively (drawing on the descriptive analytical method) to describe PPI strategies in scaling-up initiatives in HSS. We will structure the data on the included studies according to the PICOS elements and PROGRESS factors. We anticipate that the development of a quantitative meta-analysis will not be feasible due to the nature of our research question and substantial methodological heterogeneity of studies identified. If possible, we will stratify results by (1) economic status of the country (e.g., low-, middle-, or high-income country), (2) sex/gender, and (3) health context (e.g., primary care, secondary, and tertiary care). We will report our results using the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist [25], the SAGER guidelines [24], and the GRIPP2 reporting checklist [23].

Consultation with the steering committee

Our steering committee will seek consensus on a preliminary set of PPI strategies to present in an online Delphi (e-Delphi) survey. First, we will provide the committee with a list of the PPI strategies found in the literature. Each steering committee member will review the inventory and judge whether the PPI strategy represents the construct of PPI or not. Second, the project leaders will produce a comprehensive list of voted PPI strategies. Third, a videoconference will be conducted to reach a decision about which PPI strategies will be included in a preliminary list in the open round of the following Delphi study.

Delphi study

Study design

As part of the RAND/UCLA Appropriateness Method [18, 19], we will use evidence from the knowledge synthesis to achieve a consensus opinion on the relevant PPI strategies for scaling-up initiatives in HSS.

Participants

Based on sampling trends and recommendations from previous Delphi studies, we aim to invite 100 participants [46] to help ensure that a minimum of 16 stakeholders complete all the rounds [47]. We will include at least four patients or citizens (men and women), two health care providers, two policy or decision makers, two trainees, two members of patient-oriented research organizations, two scaling-up researchers, and two first or last authors of scaling-up studies. We will include them according to (1) their age (at least 18 years old); (2) their ability to participate, read, and understand English or French; and (3) their knowledge or interest in the field. We plan to recruit participants through the directed snowball method. We will ensure patients and citizens understand our research objectives and participate in decisions as we prepare and oversee information provided to them.

Data collection

We anticipate that the following three rounds will allow an acceptable degree of agreement on research priorities, but if not, a fourth round will be undertaken. In each round, we will ask participants to complete their questionnaires within 2 weeks, and we will send two reminder emails before the deadline. In order to enable efficient and timely data collection with international participants, we will use the REDCap platform, a secure web application on server infrastructure physically located at the University of British Columbia University Data Centre in Canada.

  • Round 1: We will collect participants’ sociodemographic information (e.g., country, sex, areas of expertise, primary occupational role). For each PPI strategy included in the preliminary list, we will ask participants to (1) rate how relevant the strategy is to engaging women, men, or other sex/gender categories; (2) rate how relevant the strategy is to each of the scaling-up steps; (3) determine how relevant the strategy is to various contexts (e.g., low-, middle-, or high-income countries; primary, secondary, or tertiary care); and (4) suggest any other PPI strategies required in scaling-up initiatives in HSS. Responses will be collated and discussed by our steering committee, and a list of strategies will be drawn up for scoring in the following round.

  • Round 2: Participants will be asked to individually score each PPI strategy for its overall relevance to the largest range of sex/gender categories, scaling up steps and contexts. In addition, they will be invited to provide a brief justification or cite a relevant study to support their scores. Finally, there will be an opportunity to add any further PPI strategy that respondents feel should be considered. We will collate results and calculate the median score and percentage of agreement.

  • Round 3: We will send a checklist to participants who completed the second round in which the median results from the third round are listed alongside the participant’s own score. Participants will be invited to reconsider the relevance of the PPI strategies and confirm or revise their score in the light of the participants’ opinions.

Data analysis

We will use a 10-point Likert scale ranging from 0 = “not relevant” to 9 = “highly relevant.” We will categorize participants’ responses as low (0–3), moderate (4–6), and high (7–9) [48]. For each PPI strategy, consensus will be defined as 80% agreement for the priority score of 7–9. Results will be collated using Excel to calculate the median score and percentage of agreement.

Build collaboration network

We will use a snowballing recruitment method to develop an international network of stakeholders who share an interest in the field of scaling up in HSS. The snowball method is modeled after “contact tracing” in health, in which one individual names all other individuals who were associated with a specific event [49]. We established the RePOS network in March 2020 and included over 20 members (e.g., patients, citizens, health care providers, policy makers, researchers, and trainees) from low-, middle-, and high-income countries. First, with our steering committee members, we will define eligibility criteria and draft an indicative electronic membership form. Second, we will email steering committee members, our collaborators, and all Delphi study participants to ask them to become members of the RePOS network by signing the electronic form. Also, we will ask them to suggest other stakeholders eligible for inclusion in the network. Third, we will use internet-based forums, conference calls, monthly e-bulletins, and ongoing exchange to increase the network’s reach to other experts in the field. We will continue to add stakeholders until we have no new eligible names. Finally, the creation of this group will allow us to experience all the difficulties and complexities of involving PPI and multiple stakeholders in a scaling-up project.

Discussion

In this project, we will identify relevant strategies for meaningfully and equitably involving patients and the public in the science and practice of scaling up in HSS. Overall, outcomes will be:

  1. 1)

    An inventory of existing strategies related to the involvement of patients and the public in scaling-up initiatives in HSS. This inventory will serve as a guide for those planning scaling up to map their specific PPI needs;

  2. 2)

    A consensus on most relevant strategies for PPI in scaling-up initiatives in HSS. Analysis of common elements of relevant PPI strategies will provide a generic framework for future endeavors;

  3. 3)

    An international network of stakeholders working in patient-oriented research on scaling up. Overall, this collaboration will utilize and disseminate our research results (most effective PPI strategies for scaling up evidence-based innovations) to ensure that more scaling-up initiatives worldwide reflect the needs and priorities of those they are intended to benefit.

Our project will result in a better understanding of PPI in scaling-up initiatives in HSS, a knowledge gap identified in previous work [6, 50]. Our project will also contribute to knowledge about effective PPI in other complex research projects. Our findings have the potential to increase the impact of evidence-based innovations worldwide (including in lower- and middle-income countries), ensure they benefit the people they are intended for, reduce waste, and ultimately contribute to reducing health inequities and improving the health of populations.

We acknowledge that this project includes some operational challenges, such as restricted face-to-face interactions and activities because of the coronavirus disease 2019 (COVID-19) pandemic. However, we will use teleconference software to continue working as a team and involve patients, citizens, and participants who are comfortable with using these technologies. Another potential challenge common to Delphi studies is participant dropout, but we will maintain participant enthusiasm and engagement by expressing our appreciation for their time in every email correspondence, maintaining connections in person by telephone, and following-up with non-respondents [51, 52].

Finally, we will disseminate our results to knowledge users through publications in peer-reviewed journals, presentations at relevant conferences, and organization of workshops. We will also share our reports using free public repositories such as Open Science Framework and ResearchGate. The dissemination of this project will start with the publication of the protocol. Any amendments made to this protocol will be reported in the final manuscript.

Availability of data and materials

Please send all requests for study data or materials to Dr. Ali Ben Charif (ali.bencharif@gmail.com) and Dr. France Légaré (france.legare@mfa.ulaval.ca).

Abbreviations

AHRQ:

Healthcare Research and Quality

CADTH:

Canadian Agency for Drugs and Technologies in Health

CIHR:

Canadian Institutes of Health Research

EBI:

Evidence-based innovation

GRIPP2:

Guidance for Reporting Involvement of Patients and the Public

HSS:

Health and social services

iKT:

Integrated knowledge translation

JBI:

Joanna Briggs Institute

KT:

Knowledge translation

OSF:

Open Science Framework

PICOS:

Participants, intervention, comparator, outcome, and setting

PPI:

Patient and public involvement

PPEET:

Public and Patient Engagement Evaluation Tool

PRISMA:

Preferred Reporting Items for Systematic reviews and Meta-Analyses

PRISMA-P:

Preferred Reporting Items for Systematic reviews and Meta-Analyses Protocol

PRISMA-ScR:

Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews

PRESS:

Peer Review of Electronic Search Strategies

PROGRESS:

Place of residence, race/ethnicity/culture/language, occupation, gender/sex, religion, education, socioeconomic status, and social capital

RePOS:

Research on patient-oriented scaling-up

SAGER:

Sex and Gender Equity in Research

References

  1. 1.

    Ben Charif A, Hassani K, Wong ST, Zomahoun HTV, Fortin M, Freitas A, et al. Assessment of scalability of evidence-based innovations in community-based primary health care: a cross-sectional study. CMAJ Open. 2018;6:E520–7.

    PubMed  PubMed Central  Article  Google Scholar 

  2. 2.

    Ben Charif A, Zomahoun HTV, Massougbodji J, Khadhraoui L, Pilon MD, Boulanger E, et al. Assessing the scalability of innovations in primary care: a cross-sectional study. CMAJ Open. 2020;8(4):E613–8.

  3. 3.

    Bégin M, Eggertson L, Macdonald N. A country of perpetual pilot projects. CMAJ. 2009;180(1185):E88–9.

    PubMed Central  Article  Google Scholar 

  4. 4.

    Sheridan DJ. Research: increasing value, reducing waste. The Lancet. 2014;383:1123.

    Article  Google Scholar 

  5. 5.

    McLean R, Gargani J. Scaling impact: innovation for the public good. 1st édition ed. London; New York: Routledge; 2019.

  6. 6.

    Ben Charif A, Zomahoun HTV, LeBlanc A, Langlois L, Wolfenden L, Yoong SL, et al. Effective strategies for scaling up evidence-based practices in primary care: a systematic review. Implementation Science. 2017;12:139.

    PubMed  Article  PubMed Central  Google Scholar 

  7. 7.

    Greenhalgh T, Papoutsi C. Spreading and scaling up innovation and improvement. BMJ. 2019;365:l2068.

    PubMed  PubMed Central  Article  Google Scholar 

  8. 8.

    World Health Organization (WHO). Nine steps for developing a scaling-up strategy [Internet]. 2010 [cited 2018 Apr 26]. Available from: http://www.who.int/reproductivehealth/publications/strategic_approach/9789241500319/en/

    Google Scholar 

  9. 9.

    Ghiron L, Shillingi L, Kabiswa C, Ogonda G, Omimo A, Ntabona A, et al. Beginning with sustainable scale up in mind: initial results from a population, health and environment project in East Africa. Reprod Health Matters. 2014;22:84–92.

    PubMed  Article  PubMed Central  Google Scholar 

  10. 10.

    Milat AJ, King L, Bauman AE, Redman S. The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promot Int. 2013;28:285–98.

    PubMed  Article  Google Scholar 

  11. 11.

    Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26:13–24.

    PubMed  Article  Google Scholar 

  12. 12.

    Gogovor A, Zomahoun HTV, Ben Charif A, McLean RKD, Moher D, Milat A, et al. Essential items for reporting of scaling studies of health interventions (SUCCEED): protocol for a systematic review and Delphi process. Syst Rev. 2020;9:11.

    PubMed  PubMed Central  Article  Google Scholar 

  13. 13.

    Carman KL, Dardess P, Maurer M, Sofaer S, Adams K, Bechtel C, et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff (Millwood). 2013;32:223–31.

    PubMed  Article  Google Scholar 

  14. 14.

    Pomey M-P, Flora L, Karazivan P, Dumez V, Lebel P, Vanier M-C, et al. The Montreal model: the challenges of a partnership relationship between patients and healthcare professionals. Sante Publique. 2015;27:S41–50.

    PubMed  Article  Google Scholar 

  15. 15.

    Shippee ND, Domecq Garces JP, Prutsky Lopez GJ, Wang Z, Elraiyah TA, Nabhan M, et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect. 2015;18:1151–66.

    PubMed  Article  Google Scholar 

  16. 16.

    Coulter A. Engaging patients in healthcare. UK: McGraw-Hill Education; 2011.

    Google Scholar 

  17. 17.

    Institute of Medicine (US). In: Olsen L, Saunders RS, JM MG, editors. Patients charting the course: citizen engagement and the learning health system: workshop summary [Internet]. Washington (DC): National Academies Press (US); 2011. [cited 2019 Sep 6]. Available from: http://www.ncbi.nlm.nih.gov/books/NBK91496/.

    Google Scholar 

  18. 18.

    Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA appropriateness method user’s manual [Internet]. 2001 [cited 2020 Aug 7]. Available from: https://www.rand.org/pubs/monograph_reports/MR1269.html

    Google Scholar 

  19. 19.

    van het Loo M, Kahan JP. The RAND Appropriateness method: an annotated bibliography through June 1999 [Internet]: RAND Corporation; 1999. [cited 2020 Aug 7]. Available from: https://www.rand.org/pubs/rand_europe/RE99-010.html

  20. 20.

    Canadian Institutes of Health Research (CIHR). Sex, gender and health research - CIHR [Internet]. 2018 [cited 2019 Sep 3]. Available from: http://www.cihr-irsc.gc.ca/e/50833.html

    Google Scholar 

  21. 21.

    Sudsawad P. Knowledge translation: introduction to models, strategies and measures [Internet]: Southwest Educational Development Laboratory, National Center for the …; 2007. Available from: http://www.ktdrr.org/ktlibrary/articles_pubs/ktmodels/

  22. 22.

    McMaster University. Public and Patient Engagement Evaluation Tool (PPEET) [Internet]. 2017 [cited 2020 Jun 29]. Available from: https://healthsci.mcmaster.ca/ppe/our-products/public-patient-engagement-evaluation-tool

    Google Scholar 

  23. 23.

    Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097.

    PubMed  PubMed Central  Article  Google Scholar 

  24. 24.

    Heidari S, Babor TF, De Castro P, Tort S, Curno M. Sex and Gender Equity in Research: rationale for the SAGER guidelines and recommended use. Research Integrity and Peer Review. 2016;1:2.

    PubMed  PubMed Central  Article  Google Scholar 

  25. 25.

    Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169:467–73.

    PubMed  Article  PubMed Central  Google Scholar 

  26. 26.

    Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.

    Article  Google Scholar 

  27. 27.

    Peters MDJ, Godfrey C, McInerney P, Baldini Soares C, Khalil H, Parker D. Chapter 11: scoping reviews. In: Aromataris E, Munn Z, editors. Joanna Briggs Institute Reviewer’s Manual [Internet]; 2017. [cited 2019 Sep 7]. Available from: https://reviewersmanual.joannabriggs.org/.

    Google Scholar 

  28. 28.

    Foster ED, Deardorff A. Open Science Framework (OSF). J Med Libr Assoc. 2017;105:203–6.

    PubMed Central  Google Scholar 

  29. 29.

    Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1.

    PubMed  PubMed Central  Article  Google Scholar 

  30. 30.

    Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annu Symp Proc. 2006;2006:359–63.

    PubMed Central  PubMed  Google Scholar 

  31. 31.

    Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise J-M, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012;27:985–91.

    PubMed  PubMed Central  Article  Google Scholar 

  32. 32.

    Petkovic J, Riddle A, Akl EA, Khabsa J, Lytvyn L, Atwere P, et al. Protocol for the development of guidance for stakeholder engagement in health and healthcare guideline development and implementation. Syst Rev. 2020;9:21.

    PubMed  PubMed Central  Article  Google Scholar 

  33. 33.

    Tugwell P, Robinson V, Grimshaw J, Santesso N. Systematic reviews and knowledge translation. Bull World Health Organ. 2006;84:643–51.

    PubMed  PubMed Central  Article  Google Scholar 

  34. 34.

    Nicholson K, Ganann R, Bookey-Bassett S, Garland Baird L, Garnett A, Marshall Z, et al. Capacity building and mentorship among pan-Canadian early career researchers in community-based primary health care. Prim Health Care Res Dev. 2020;21:e3.

    PubMed  PubMed Central  Article  Google Scholar 

  35. 35.

    Dukhanin V, Topazian R, DeCamp M. Metrics and evaluation tools for patient engagement in healthcare organization- and system-level decision-making: a systematic review. Int J Health Policy Manage. 2018;7:889–903 Kerman University of Medical Sciences.

    Article  Google Scholar 

  36. 36.

    The Lancet. What is health? The ability to adapt. Lancet. 2009;373:781 Elsevier.

    Article  Google Scholar 

  37. 37.

    McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

    PubMed  Article  Google Scholar 

  38. 38.

    Agency for Healthcare Research and Quality (AHRQ). Methods guide for effectiveness and comparative effectiveness reviews [Internet]. Rockville: Agency for Healthcare Research and Quality (US); 2008. [cited 2019 Aug 28]. Available from: http://www.ncbi.nlm.nih.gov/books/NBK47095/

    Google Scholar 

  39. 39.

    Porterfield DS, Hinnant LW, Kane H, Horne J, McAleer K, Roussel A. Linkages between clinical practices and community organizations for prevention: a literature review and environmental scan. American journal of public health. 2012;102:S375–82.

    PubMed  PubMed Central  Article  Google Scholar 

  40. 40.

    Taymour RK, Abir M, Chamberlin M, Dunne RB, Lowell M, Wahl K, et al. Policy, practice, and research agenda for emergency medical services oversight: a systematic review and environmental scan. Prehosp Disaster Med. 2018;33:89–97.

    PubMed  Article  PubMed Central  Google Scholar 

  41. 41.

    International Conference on Grey Literature, Commission of the European Communities, Directorate-General Telecommunications IM and Exploitation of Research, editors. Perspectives on the design and transfer of scientific and technical information: third International Conference on Grey Literature, Jean Monnet Building, Luxembourg, 13-14 November 1997 : GL’97 proceedings. Amsterdam, The Netherlands: TransAtlantic GreyNet; 1998.

    Google Scholar 

  42. 42.

    Canadian Agency for Drugs and Technologies in Health (CADTH). Grey matters: a practical tool for searching health-related grey literature [Internet]. Ottawa: CADTH; 2018. [cited 2021 Jan 29]. Available from: https://www.cadth.ca/resources/finding-evidence.

  43. 43.

    Veritas Health Innovation. Covidence systematic review software [Internet]. [cited 2020 Mar 11]. Available from: www.covidence.org.

  44. 44.

    O’Neill J, Tabish H, Welch V, Petticrew M, Pottie K, Clarke M, et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol. 2014;67:56–64.

    PubMed  Article  PubMed Central  Google Scholar 

  45. 45.

    Tricco AC, Zarin W, Rios P, Pham B, Straus SE, Langlois EV. Barriers, facilitators, strategies and outcomes to engaging policymakers, healthcare managers and policy analysts in knowledge synthesis: a scoping review protocol. BMJ Open. 2016;6:e013929 British Medical Journal Publishing Group.

    PubMed  PubMed Central  Article  Google Scholar 

  46. 46.

    Boulkedid R, Schroedt J, Mathoulin Pelissier S, Boizeau P, Alberti C. Combien faut-il d’experts dans une enquête Delphi? Étude de simulation. Revue d’Épidémiologie et de Santé Publique. 2017;65:S74.

    Article  Google Scholar 

  47. 47.

    Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41:376–82.

    PubMed  Article  PubMed Central  Google Scholar 

  48. 48.

    Pinnock H, Epiphaniou E, Sheikh A, Griffiths C, Eldridge S, Craig P, et al. Developing standards for reporting implementation studies of complex interventions (StaRI): a systematic review and e-Delphi. Implement Sci. 2015;10:42.

    PubMed  PubMed Central  Article  Google Scholar 

  49. 49.

    Sadler GR, Lee H-C, Seung-Hwan Lim R, Fullerton J. Recruiting hard-to-reach United States population sub-groups via adaptations of snowball sampling strategy. Nurs Health Sci. 2010;12:369–74.

    PubMed  PubMed Central  Article  Google Scholar 

  50. 50.

    Zomahoun HTV, Ben Charif A, Freitas A, Garvelink MM, Menear M, Dugas M, et al. The pitfalls of scaling up evidence-based interventions in health. Glob Health Action. 2019;12:1670449.

    PubMed  PubMed Central  Article  Google Scholar 

  51. 51.

    Nguyen DTK. Scaling up [to] a population health intervention: a readiness assessment framework: Graduate Studies; 2016. [cited 2020 Jun 23]; Available from: https://prism.ucalgary.ca/handle/11023/2955

  52. 52.

    Hung H-L, Altschuld JW, Lee Y-F. Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. Evaluation and Program Planning. 2008;31:191–8.

    PubMed  Article  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank Louisa Blair, English-language editor, for her kind help with the manuscript. We have prepared this work on behalf of the Research on Patient-Oriented Scaling-up (RePOS) Network, which includes at the date of submission of this article the following stakeholders:

Last name, First name Country Sex Type of stakeholder Email
Beaumont, Martin Canada Male Policy maker martin.beaumont@chudequebec.ca
Beleno, Ron Canada Male Patient instructor ron@rb33.com
Ben Charif, Ali Comoros, Canada Male Trainee, investigator ali.bencharif@gmail.com
Bierman, Arlene USA Female Policy maker arlene.bierman@ahrq.hhs.gov
Blais, Johanne Canada Female Provider blaisj2@videotron.ca
Fancott, Carol Canada Female Decision maker, investigator carol.fancott@cfhi-fcass.ca
Geiger, Friedemann Germany Male Provider, investigator f.geiger@uksh.de
Gogovor, Amédé Togo, Canada Male Trainee, investigator amede.gogovor@gmail.com
Guay-Bélanger, Sabrina Canada Female Investigator sabrina.guay-belanger.ciussscn@ssss.gouv.qc.ca
Kastner, Kathy Canada Female Patient instructor kathyk@bestendings.com
Légaré, France Canada Female Provider, investigator france.legare@mfa.ulaval.ca
McLean, Robert Canada, South Africa Male Policy maker, investigator rmclean@idrc.ca
Milat, Andrew J Australia Male Policy maker, investigator andrew.milat@health.nsw.gov.au
Paquette, Jean-Sébastien Canada Male Provider, investigator jspaquette.lab@gmail.com
Plourde, Karine V. Canada Female Investigator karine.plourde2.ciussscn@ssss.gouv.qc.ca
Rivest, Francois Canada Male Investigator francois.rivest@canada.ca
Straus, Sharon Canada Female Provider, investigator sharon.straus@utoronto.ca
Thibodeau, Guy Canada Male Decision maker guy.thibodeau.ciussscn@ssss.gouv.qc.ca
Wolfenden, Luke Australia Male Investigator luke.wolfenden@hnehealth.nsw.gov.au
Zomahoun, Hervé Tchala Vignon Benin, Canada Male Investigator herve.zomahoun.ciussscn@ssss.gouv.qc.ca

Funding

Our project will be funded by the following Canadian Institutes of Health Research (CIHR) grants: (1) a Catalyst Grant (#PAO-169411), (2) a Foundation Grant (#FDN-159931), and (3) the Quebec Strategy for Patient-Oriented Research (SPOR) Support for People and Patient-Oriented Research and Trials (SUPPORT) Unit (#SU1-139759). ABC is supported by the Fonds de recherche en santé du Québec–Santé (FRQ-S). AG is supported by the CIHR through the Patient-Oriented Research Awards–Transition to Leadership Stream. LW is funded by a NHMRC Career Development and Heart Foundation Future Leaders Fellowship. FL holds a Tier 1 Canada Research Chair in Shared Decision Making and Knowledge Translation (#950-231200). The funding agreement ensures the authors’ independence in designing the study, writing, and publishing this article. The information provided or views expressed in this article are the responsibility of the authors alone.

Author information

Affiliations

Authors

Consortia

Contributions

ABC, KVP, SGB, HTVZ, AG, and FL participated in the conception of this project. ABC, KVP, SGB, RB, KK, and FL participated in the design and implementation of this project. ABC drafted the manuscript. All authors revised the manuscript critically for important intellectual content, gave final approval of the version to be published, and agreed to be accountable for all aspects of the project.

Corresponding author

Correspondence to France Légaré.

Ethics declarations

Ethics approval and consent to participate

Ethical approval for this project was obtained from the Ethics Board of the Centre intégré universitaire de santé et de services sociaux de la Capitale-Nationale (CIUSSS-CN). All studies within this project will be designed and performed in accordance with the Declaration of Helsinki. Electronic and written informed consent will be obtained from all the participants of the Delphi study.

Consent for publication

Not applicable.

Competing interests

The authors have none to declare.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

PRISMA-P 2015 Checklist.

Additional file 2.

Search strategy.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ben Charif, A., Plourde, K.V., Guay-Bélanger, S. et al. Strategies for involving patients and the public in scaling-up initiatives in health and social services: protocol for a scoping review and Delphi survey. Syst Rev 10, 55 (2021). https://doi.org/10.1186/s13643-021-01597-6

Download citation

Keywords

  • Patient and public involvement
  • Scaling up
  • Spread
  • Scaling science
  • Health and social services
  • Implementation science
  • Knowledge translation
  • Strategy