- Open Access
Improving the usefulness of evidence concerning the effectiveness of implementation strategies for knowledge products in primary healthcare: protocol for a series of systematic reviews
Systematic Reviews volume 9, Article number: 112 (2020)
The literature on the implementation of knowledge products is extensive. However, this literature is still difficult to interpret for policymakers and other stakeholders when faced with choosing implementation strategies likely to bring about successful change in their health systems. This work has the particularity to examine the scope of this literature, and to clarify the effectiveness of implementation strategies for different knowledge products. Consequently, we aim to (1) determine the strengths and weaknesses of existing literature overviews; (2) produce a detailed portrait of the literature on implementation strategies for various knowledge products; and (3) assess the effectiveness of implementation strategies for each knowledge product identified and classify them.
We will use a three-phase approach consisting of a critical analysis of existing literature overviews, a systematic review of systematic reviews, and a series of systematic reviews and meta-analyses. We will follow the Cochrane Methodology for each of the three phases. Our eligibility criteria are defined following a PICOS approach: Population, individuals or stakeholders participating in healthcare delivery, specifically, healthcare providers, caregivers, and end users; Intervention, any type of strategy aiming to implement a knowledge product including, but not limited to, a decision support tool, a clinical practice guideline, a policy brief, or a decision-making tool, a one-pager, or a health intervention; Comparison, any comparator will be considered; Outcomes, phases 1 and 2—any outcome related to implementation strategies including, but not limited to, the measures of adherence/fidelity to the use of knowledge products, their acceptability, adoption, appropriateness, feasibility, adaptability, implementation costs, penetration/reach and sustainability; phase 3—any additional outcome related to patients (psychosocial, health behavioral, and clinical outcomes) or healthcare professionals (behavioral and performance outcomes); Setting, primary healthcare has to be covered. We will search MEDLINE (Ovid), EMBASE, Web of Science, PsycINFO, CINAHL, and the Cochrane Library from their inception onwards. For each phase, two reviewers will independently perform the selection of studies, data extraction, and assess their methodological quality. We will analyze extracted data, and perform narrative syntheses, and meta-analyses when possible.
Our results could inform not only the overviews’ methodology but also the development of an online platform for the implementation strategies of knowledge products. This platform could be useful for stakeholders in implementation science.
Systematic review registration
Protocol registered on Open Science Framework, https://osf.io/eb8w2/.
Description of the condition
Having an evidence-based knowledge product does not mean that it will be systematically implemented with success and be beneficial for patients. Balas et al.  estimated that it takes up to 17 years to translate 14% of original research into practices for the benefit of patients. Given this gap between the production of health evidence and its application in routine practice, increasing attention has been devoted to knowledge translation (KT), and more specifically, to KT [2, 3] and implementation strategies [3,4,5]. Knowledge products considered in implementation strategies include clinical practice guidelines, shared decision-making tools, decision support tools, policy briefs, a one-pager (simple, iconographic, infographic), and a health intervention (technological, pharmacological, behavioral, or management). As a result, there has been a substantial increase in studies examining their effectiveness.
Description of the implementation
Implementation can be defined as an actively planned and deliberatively initiated effort with the intention to translate a given knowledge product into practices in a particular setting and context [6, 7]. Implementation requires characterizing the knowledge product to be implemented as well as the context and setting of implementation . It is also important to (1) determine an adapted process framework or theory; (2) clarify the different stages of the implementation process; (3) identify stakeholders to be involved; (4) recognize facilitators and barriers to implementation using an appropriate framework; (5) identify the corresponding implementation strategies; and (6) target the relevant outcomes and tools to evaluate them . Implementation strategies have been explored extensively in the literature. They can be defined as techniques or methods aiming to improve or optimize the uptake and routine application of complex interventions in clinical care . To be better characterized, implementation strategies have to be named, defined, and specified . The Cochrane Effective Practice and Organization of Care (EPOC) grouped existing implementation strategies in categories targeting respectively healthcare organizations, healthcare professionals, specific types of practice, condition, or setting .
How the knowledge product implementation might work
The implementation of a given knowledge product should be guided by one or more frameworks/models/theories depending on its complexity. As suggested by Nilsen , an implementation process, identification of its determinants, and evaluation of its outcomes may be supported by a framework/model/theory. Indeed, frameworks/models/theories permit stakeholders to better explain and clarify different steps of the implementation process, i.e., a description of the main steps to translate a given knowledge product into practices followed by the evaluation of outcomes and the sustainability of its use (e.g., the knowledge to action framework ). Frameworks/models/theories can also be used to better address the various levels of factors that potentially influence implementation outcomes (e.g., the Consolidated Framework for Implementation Research, CFIR ). Finally, there are frameworks/models/theories that have been developed to optimally address the various levels of outcomes that can both explain how different steps of the implementation process took place as well as indicate the timeline for their evaluation [10, 11].
Why it is important to do a series of systematic reviews
According to our preliminary literature search, we found that the literature on the implementation of knowledge products is large. In an effort to further synthesize, several overviews of these systematic reviews [4, 12,13,14,15,16,17,18,19,20,21] have been published in the last 20 years. Overviews attempt to systematically retrieve and summarize the results of multiple systematic reviews on the effectiveness of interventions or treatments for a given condition or public health problem . Their purpose is to contrast the findings of several systematic reviews on the same subject, thus offering decision-makers a broader understanding of the evidence. While, in theory, this type of study has the potential to overview the evidence, its execution is fraught with several challenges. There are guidelines for conducting this type of research [23,24,25,26]; however, several experts who have examined the methodology regarding overviews of systematic reviews over the past decade agreed that existing directives and recommendations are not sufficiently robust and there is great room for improvement [27,28,29].
This literature remains difficult to interpret for policymakers and other stakeholders when faced with having to select implementation strategies likely to bring about successful change in their health systems. Indeed, it remains a challenge to determine what strategy is effective for the implementation of a given knowledge product. Examining the scope of this literature and further clarifying the effectiveness of implementation strategies for different knowledge products could serve to identify original and relevant research questions for future systematic reviews. In turn, these latter may allow us to quantitatively assess the effectiveness of implementation strategies for specific knowledge products. The general aim of this project is to identify, for each category of knowledge product, the most effective implementation strategies for their uptake into healthcare professionals’ clinical practice.
We aim to address the following specific objectives in three distinct phases:
Phase 1—to perform a critical analysis of existing literature overviews to determine their strengths and weaknesses. This analysis will not only help us better define our research question for subsequent phases but will also highlight many of the methodological challenges we may encounter.
Phase 2—to conduct a systematic review of systematic reviews while taking into account the weaknesses of previously published overviews, and provide a detailed portrait of the literature on various implementation strategies for knowledge.
Phase 3—to perform a series of meta-analyses that evaluate the effectiveness of implementation strategies for the different types of knowledge products, using individual randomized controlled trials (RCTs) from systematic reviews included in phase 2. These analyses will provide answers to specific research questions with the most robust experimental study designs, and more detailed ones, in which contextual elements, often absent in overviews, are going to be taken into consideration.
This study protocol has been registered within the Open Science Framework platform (Registration ID: https://osf.io/eb8w2/). We are reporting it in accordance with the reporting guidance provided in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) statement (See PRISMA-P checklist in Additional file 2) . We will conduct this project following the recommendations in the Cochrane Handbook for systematic reviews .
We will use a three-phase approach (see Fig. 1 for details) including a critical analysis of existing literature overviews, a systematic review of systematic reviews, and a series of systematic reviews and meta-analyses.
In the first two phases of the project, according to a PICOS (Population, Intervention, Comparison, Outcomes, Study design) with knowledge products, studies that meet the following criteria will be considered for inclusion:
The population of interest is individuals or stakeholders participating in healthcare delivery, specifically, healthcare providers, caregivers, and end users (i.e., patients or clients with any conditions).
We will consider any type of strategy aiming to implement a knowledge product. The EPOC taxonomy will be used to classify implementation strategies identified (e.g., audit and feedback, educational meetings, interprofessional education) .
As suggested by Pinnock et al. , a clear distinction has been made between the implementation strategies and the knowledge products implemented. A knowledge product could be, but not limited to, a decision support tool, a clinical practice guideline, a policy brief, or a decision-making tool, a one-pager (simple, iconographic, infographic), or a health intervention (technological, pharmacological, behavioral, or management). Only implementation studies in which both are reported will be considered in our project.
Any comparator will be considered. This includes studies in which two or more implementation strategies for knowledge products are compared (head to head); and studies in which there is no comparator.
In phases 1 and 2, we will mainly consider outcomes related to implementation strategies including, but not limited to, the measures of adherence/fidelity to the use of knowledge products, their acceptability, adoption, appropriateness, feasibility, adaptability, implementation costs, penetration/reach, and sustainability . As for phase 3, we will additionally consider the outcomes related to patients (psychosocial, health behavioral, and clinical outcomes) or healthcare professionals (psychosocial, health behavioral, and performance outcomes) that the knowledge products aim to improve.
This eligibility criterion will be specific to each phase of the project:
Phase 1. We will consider any review of reviews on implementation strategies for knowledge products (i.e., a knowledge synthesis in which included studies are any literature reviews).
Phase 2. We will consider only systematic reviews (i.e., any study in which authors performed a narrative and/or quantitative synthesis of experimental studies on implementation strategies for knowledge products using a comprehensive and reproducible approach).
Phase 3. Only RCTs will be considered.
Any health domain addressed in primary healthcare will be considered. A review will be included if primary healthcare is covered.
Information sources and search strategy
We will perform comprehensive searches within five electronic databases: MEDLINE (Ovid), EMBASE, Web of Science, PsycINFO, CINAHL, and the Cochrane Library from their inception onwards. The search strategy will be designed by an information specialist using “knowledge transfer” and “implementation strategies” headings (See draft search strategy for MEDLINE/Ovid in Additional file 1). For efficiency considerations, we will not build a new search strategy to identify RCTs, rather, we plan in identifying them from the systematic reviews that we will select in our overview. However, to avoid missing any recently published RCTs in the field, we will update our search strategy by looking for primary studies published after the most recent included reviews. Searches will be conducted within the same electronic databases as in earlier phase. Study designs other than RCTs, for example systematic reviews, quasi-experimental studies and observational studies, will be excluded.
There will be neither language nor literature search date restrictions. The references listed in all eligible studies will also be manually searched in order to identify additional relevant ones.
We will merge the citations identified from five electronic databases mentioned above in the EndNote software. Then, we will identify and remove the duplicates. The unique citations will be considered for the selection process.
Selection of studies
Two assessors will independently contribute to all steps of the selection process. References identified from relevant electronic databases will be merged, and duplicates removed to obtain a database including unique citations for the study selection process. Assessors will discuss the inclusion criteria to ensure mutual understanding, and pilot test on five percent of unique citations identified to confirm that the evaluation process is reliable. The pilot section will be considered conclusive if the kappa statistic referring to the agreement between assessors is greater than 0.7 .
Pairs of assessors will then independently screen for titles and abstracts based on inclusion criteria specific to each phase outlined above. In case of doubt, the citations will be included and considered for full-text reading. Assessors will then independently screen the full texts of references retained at the first step. Any disagreement will be resolved by consensus or by involving a third assessor.
Assessors will include overviews for phase 1 and systematic reviews for phase 2, based on the eligibility criteria mentioned above. For phase 3, the following procedure will be adopted for the identification of relevant RCTs from each review considered: first, we will go through the list including primary studies to identify RCTs that are clearly reported as such by the review’s authors. In the case, RCTs are not, we will select the complete list of primary articles included. Removal of duplicates and screening based only on the study design will be done by two independent assessors.
For each phase, a pilot data extraction will be done on 5% of selected articles. The data extraction will be pursued only when the pilot is conclusive. This process will be independently performed by two assessors. Discrepancies between them will be resolved by a third author.
The following data will be extracted: the first author’s name; the year of publication; the focus of the overview, based on the scope of PICO elements; the number and name of databases consulted; the period of the literature search; the strategies used to update the literature; the type and number of reviews that are included; the type of reviews (Cochrane, non-Cochrane, both); the strategies to deal with the overlap between reviews; the strategies to deal with conflicting results of reviews; the quality assessment of reviews; the type of synthesis performed (narrative, meta-analysis or both); the assessment of the quality of evidence, and the limitations reported by overview authors.
We will extract data concerning the following elements:
Review details. First author’s name, year of publication, type, objectives, registration information.
Literature search details. Number and names of electronic databases, search for gray literature, search period, restrictions, design and number of primary studies.
Characteristics of participants. Name, profile, total number, age (mean, median, and/or range), gender (percentage of men or women).
Characteristics of implementation strategies. Components described according to the EPOC taxonomy, knowledge products implemented (e.g., clinical guidelines, health intervention).
Characteristics of outcomes. Name, type of measurement tools used (self-administered, objective, both). We will use Proctor et al. taxonomy to guide the classification of outcomes .
Assessment of study quality. Name of the tool used, result.
Type of synthesis. Narrative, meta-analysis, or both.
Assessment of publication bias. Method used (funnel plots, statistical tests, both), and treatment of it if present.
Assessment of the quality of evidence. Name of the tool used, level of evidence.
Conclusion of reviews. Information reported by authors when they conclude (e.g., main results and limitations).
For each RCT considered in this phase, we will extract data concerning the following elements:
Characteristics of the study. First author’s name, year of publication, country, language, healthcare domain, setting, study design, sampling, and recruitment method.
Characteristics of participants. Size of eligible population for both healthcare providers and end users, total sample size, sample sizes in implementation strategy, and control groups, unit of allocation, mean age or range, sex, race/ethnicity (percentage of Caucasians), socioeconomic level of the sampled population, and profile.
Characteristics of implementation strategies (description of the intervention). Components (according to the EPOC taxonomy), underlying framework, content, number of participating centers, mode of delivery (e.g., in person and online), actual duration (e.g., number of hours), actual frequency (e.g., number of sessions), methods used (e.g., interactive or didactic techniques, web applications), course materials used (e.g., web capsules, textbooks), adherence/fidelity, and content of the intervention offered to control group.
Characteristics of knowledge products implemented. Name, format and content.
Characteristics of outcomes and analysis methods used. Names of interest outcomes, measurement tools, range of scale and analysis methods used, unit of analysis, intention to treat analysis or not, retention rates, and effect measures of implementation strategies.
Effects of implementation strategies. Any information useful to estimate the effect size and its 95% confidence interval will be extracted. For continuous outcomes, we will consider the standardized mean difference as effect size requiring the sample size, mean score, and standard deviation in each group studied. For dichotomous outcomes, we will consider the odds ratio as effect size requiring the sample size and number of events in each group studied. We will also collect the value of effect size (crude and adjusted) and its confidence interval estimated in the studies. Information about confounding factors will as well be extracted.
When necessary, corresponding or first authors will be contacted to obtain information about missing data in their studies.
Quality assessment of studies
The quality of overviews retrieved in phase 1 will not be assessed since there is no methodological tool designed for this purpose. Moreover, the objective of this phase is not to draw any conclusions on the effectiveness of implementation strategies from these overviews.
For phase 2, we will assess the methodological quality of selected systematic reviews using the AMSTAR 2 tool . Those meeting the criteria of good or medium quality will be retained for the rest of the process, i.e., will be considered for the next steps of the review of reviews and for the identification of RCTs in phase 3.
As for phase 3, RCTs extracted from systematic reviews will be assessed with the Cochrane Risk of Bias (RoB) 2.0 tool .
Both the quality assessment of systematic reviews as the assessing risk of bias for RCTs will be performed independently by two authors. In a pilot assessment of two or four studies included, they will agree on a common understanding of the definitions, criteria, and guidelines provided in the tool to achieve a more objective assessment. The assessment of all studies included will do once the pilot test is conclusive. When needed, a third reviewer will be invited to help reach consensus.
We will perform a descriptive analysis using counts (number and percentage) to summarize data collected followed by carrying out a critical analysis on the latter considering the following methodological elements: literature search, methodological limitations reported in overviews, overlap between reviews included in overviews, types of data synthesis, quality of reviews included in overviews, and level of evidence.
We will produce a descriptive Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart of the selection process for all included reviews . We will then describe the characteristics of the latter, populations, implementation strategies, and outcomes assessed. We will also analyze the data on publication bias and quality of evidence assessment. Finally, evaluation data of included reviews from the AMSTAR 2 tool  will be presented using a graph and table.
First, we will classify RCTs according to the type of knowledge product being implemented. These may include but not limited to clinical guidelines, decision support tools, research summaries, one-pagers (simple, iconographic, infographic), or other health interventions (technological, pharmacological, behavioral, or management).
Second, we will present a descriptive flowchart of the RCTs included in each meta-analysis, according to the PRISMA guidelines . We will describe by frequencies (number and percentage) the characteristics of studies, populations, implementation strategies, and outcomes. Risk of bias levels will also be described for all included RCTs.
Finally, for each outcome and knowledge product, we will determine if there are sufficient data to perform a meta-analysis. If not, we will conduct a narrative synthesis on the effects of implementation strategies, and include in tables and figures to aid in data presentation. If so, we will do a meta-analysis. For each outcome and knowledge product, the random effect model will be used to estimate the pooled effect size of an implementation strategy and its 95% confidence interval, as we anticipate heterogeneity among the RCTs concerning the types of implementation intervention and population [31, 37]. For dichotomous data, the effect size will be expressed as a risk ratio or odds ratio; for continuous data, it will be provided as a standardized mean difference if different measures are used for the same outcome. For cluster RCTs, the analysis will be adjusted for clustering to avoid unit-of-analysis errors . The influence of clinical and methodological heterogeneity on the observed effects will be discussed. Subgroup analyses will be carried out if necessary according to characteristics of the studies, the populations and the implementation interventions mentioned above. Statistical heterogeneity will be assessed using the Higgins’ I square statistic [38, 39]. A funnel plot will be generated to assess publication bias if 10 or more studies are included in the meta-analysis . Statistical tests for funnel plot asymmetry (e.g., Egger’s regression, Begg’s test, and Harbord’s test) will be performed where appropriate [31, 40].
Sensitivity analyses will be carried out excluding the RCTs with high risk of bias from pooled effect size estimates. For each outcome and knowledge product, we will also explore the influence of each RCT by removing its individual effect size from the pooled estimation. These different analyses will allow us to evaluate the robustness of our results.
Assessment of the quality of evidence
The quality of evidence for each outcome and knowledge product will be assessed with the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) to reduce the misinterpretation of our review’s results . The GRADE tool is based on five criteria for each individual study, namely risk of bias, indirectness of evidence, data heterogeneity, imprecision of effect size estimates, and risk of publication bias . Overall assessment will be rated very low, low, moderate, and high for each outcome .
Our project aims to improve the usefulness of evidence on implementation strategies for knowledge products in primary healthcare. Our expected findings will have both methodological and practice implications. First, findings from the present project, mainly the ones related to the critical analysis on methodology, will contribute to methodological aspects for designing and conducting literature overviews. To this end, we will identify methodological limitations in overviews, and formulate corresponding recommendations with our collaborative team of methodologists. These recommendations will be applied to our own overview in order to determine their applicability. Second, our findings will also serve to provide a list of implementation strategies for various knowledge products. This list will include established effective implementation strategies, those potentially effective, and the non-effective ones for each knowledge product identified. These findings will inform the development of an online platform on implementation strategies for knowledge products in primary healthcare.
Apart from the methodological challenges already taken into account in our methods section, we anticipate other limitations in the realization of our project. First, there may be discrepancies between what we have planned in this protocol and what we will finally do. So, we plan for justifying them and documenting important protocol amendments. Second, doing a series of systematic reviews can take a long time, so it could be challenging to keep update the literature search of interest. To save time, we plan to create parallel review teams that will simultaneously work at the steps of study selection, data extraction, and/or study quality assessment for the critical analysis of existing overviews and the systematic reviews of systematic reviews. For each systematic review of RCTs on specific research questions, we plan to systematically update our literature search to avoid missing any recently published RCTs.
We will use passive and active strategies to disseminate our findings.
Findings from this project will be published in leading journals in the field intended for knowledge users and stakeholders in implementation science. We plan to produce a series of manuscripts for publication in peer-reviewed journals organized in the following:
A manuscript as a preamble in which we will report results from our critical analysis of existing overviews
A paper describing results from our overview and our original methodology
At least four manuscripts corresponding to each specific meta-analysis or narrative synthesis we will perform. In these papers, we will identify effective implementation strategies for each type of knowledge product
The findings will also be presented at local, national, and international conferences (number equal to four or six). Specialized conferences whose themes related to knowledge translation in health, implementation science, or primary healthcare will be privileged (e.g., the Annual Scientific Meeting of Knowledge Translation Canada, the Annual Conference on the Science of Dissemination and Implementation in Health, or the Annual Meeting of North American Primary Care Research Group). In these cases, we will solicit our expert partners on knowledge translation to participate as credible messengers. Finally, we will design a database on our website (http://unitesoutiensrapqc.ca/) on the effectiveness of implementation strategies stratified according to the types of outcomes and knowledge products. This database will be useful for anyone who wants to design an implementation study on a knowledge product.
Findings from this project are of critical relevance to each of the following knowledge platforms or teams: the Knowledge Translation Components of Strategy for Patient-Oriented Research (SPOR) - Support for People and Patient-Oriented and Trials (SUPPORT) Units, the Réseau provincial de recherche en adaptation-réadaptation (REPAR), and the Centre de recherche interdisciplinaire en réadaptation (CRIR). One of the mandates from these organizations is to guide the application of evidence-based knowledge in primary care clinical practices, as well as in rehabilitation. Since our findings will be the product of a systematic approach, we will integrate them into the range of services we offer, including training workshops (e.g., for conferences or as interactive online training with a follow-up) targeting specific audiences such as health administrators/decision-makers/health professionals and patients. We will also develop a platform on implementation strategies for knowledge products in primary healthcare. A video capsule in plain language will be created to support the use of this platform and to train the potential users including healthcare professionals, health system decision-makers, patients, and researchers involving in implementation science. We will use various communication channels to publicize our platform: social media (e.g., Facebook, Twitter), our bimonthly newsletter (more than 500 subscribers), and clients’ emails through the database, particularly those who accepted to be contacted for our advertisements.
Availability of data and materials
All data and materials used at this step are available in our protocol. There are currently no additional data or materials. However, all data and materials used during the review will be available from the corresponding author.
A MeaSurement Tool to Assess systematic Reviews
Consolidated Framework for Implementation Research
Cumulative Index to Nursing and Allied Health Literature
Centre de recherche interdisciplinaire en réadaptation
Excerpta Medica dataBASE
Effective Practice and Organization of Care
Grading of Recommendations, Assessment, Development and Evaluation
Medical Literature Analysis and Retrieval System Online
Réseau provincial de recherche en adaptation-réadaptation
Randomized controlled trials
Population, Intervention, Comparison, Outcomes
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols
Strategy for Patient-Oriented Research
Support for People and Patient-Oriented and Trials
Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD. Improving preventive care by prompting physicians. Arch Intern Med. 2000;160(3):301–8.
LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12:751.
Effective Practice and Organisation of Care (EPOC). EPOC taxonomy. 2015.
Pantoja T, Opiyo N, Lewin S, Paulsen E, Ciapponi A, Wiysonge CS, et al. Implementation strategies for health systems in low-income countries: an overview of systematic reviews. Cochrane Database Syst Rev. 2017;9:Cd011086.
Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.
Pfadenhauer LM, Gerhardus A, Mozygemba K, Lysdahl KB, Booth A, Hofmann B, et al. Making sense of complexity in context and implementation: the context and implementation of complex interventions (CICI) framework. Implement Sci. 2017;12(1):21.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53.
Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24.
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.
Brouwers MC, Garcia K, Makarski J, Daraz L. The landscape of knowledge translation interventions in cancer control: what do we know and where to next? A review of systematic reviews. Implement Sci. 2011;6:130.
Chan WV, Pearson TA, Bennett GC, Cushman WC, Gaziano TA, Gorman PN, et al. ACC/AHA special report: clinical practice guideline implementation strategies: a summary of systematic reviews by the NHLBI implementation science work group: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. J Am Coll Cardiol. 2017;69(8):1076–92.
Cheung A, Weir M, Mayhew A, Kozloff N, Brown K, Grimshaw J. Overview of systematic reviews of the effectiveness of reminders in improving healthcare professional behavior. Syst Rev. 2012;1:36.
Durieux P, Ravaud P, Dosquet P, Durocher A. Effectiveness of clinical guideline implementation strategies: systematic review of systematic reviews. Gastroenterologie clinique et biologique. 2000;24(11):1018–25.
Durieux PR, P. Efficacité des méthodes de mise en œuvre des recommandations médicales. Haute Autorité de Santé, professionnelles Sdbp; 2014.
Goldner EM, Jenkins EK, Fischer B. A narrative review of recent developments in knowledge translation and implications for mental health care providers. Canadian Journal of Psychiatry Revue Canadienne de Psychiatrie. 2014;59(3):160–9.
Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior: an overview of systematic reviews of interventions. Medical Care. 2001;39(8 Suppl 2):Ii2–45.
Lau R, Stevenson F, Ong BN, Dziedzic K, Treweek S, Eldridge S, et al. Achieving change in primary care--effectiveness of strategies for improving implementation of complex interventions: systematic review of reviews. BMJ Open. 2015;5(12):e009993.
Lewin S, Lavis JN, Oxman AD, Bastias G, Chopra M, Ciapponi A, et al. Supporting the delivery of cost-effective interventions in primary health-care systems in low-income and middle-income countries: an overview of systematic reviews. Lancet (London, England). 2008;372(9642):928–39.
Prior M, Guerin M, Grimmer-Somers K. The effectiveness of clinical guideline implementation strategies--a synthesis of systematic review findings. Journal of evaluation in clinical practice. 2008;14(5):888–97.
Pieper D, Buechter R, Jerinic P, Eikermann M. Overviews of reviews often have limited rigor: a systematic review. Journal of clinical epidemiology. 2012;65(12):1267–73.
Becker LA OA. Chapter 22: Overviews of reviews. Higgins JPT, Green S (editors), Cochrane Handbook for Systematic Reviews of Interventions. Chichester (UK): John Wiley & Sons; 2008.
Biondi-Zoccai G. Umbrella reviews evidence synthesis with overviews of reviews and meta-epidemiologic studies. Cham: Springer; 2016. Available from: Accès via SpringerLink http://dx.doi.org/https://doi.org/10.1007/978-3-319-25655-9 http://ariane.ulaval.ca/cgi-bin/recherche.cgi?qu = a2547826.
Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. International journal of evidence-based healthcare. 2015;13(3):132–40.
Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011;11(1):15.
Pieper D, Antoine SL, Morfeld JC, Mathes T, Eikermann M. Methodological approaches in conducting overviews: current state in HTA agencies. Research synthesis methods. 2014;5(3):187–99.
Pieper D, Pollock M, Fernandes RM, Buchter RB, Hartling L. Epidemiology and reporting characteristics of overviews of reviews of healthcare interventions published 2012-2016: protocol for a systematic review. Systematic reviews. 2017;6(1):73.
McKenzie JE, Brennan SE. Overviews of systematic reviews: great promise, greater challenge. Syst Rev. 2017;6(1):185.
Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1.
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA. Cochrane handbook for systematic reviews of interventions version 6.0 (Updated July 2019). Cochrane Available from wwtrainingcochraneorg/handbook. 2019.
Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for reporting implementation studies (StaRI): explanation and elaboration document. BMJ open. 2017;7(4):e013318.
Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977;33(2):363–74.
Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358:j4008.
Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ (Clinical research ed). 2019;366:l4898.
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ (Clinical research ed). 2009;339:b2700.
Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to meta-analysis: John Wiley & Sons; 2011.
Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.
Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ (Clinical research ed). 2003;327(7414):557–60.
Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ (Clinical research ed). 2011;343:d4002.
Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ (Clinical research ed). 2008;336(7650):924–6.
We thank Mr. Dominique Fournier, a scientific English language editor, for his kind help with the manuscript.
This review is funded by the Quebec Strategy for Patient-Oriented Research (SPOR) - Support for People and Patient-Oriented and Trials (SUPPORT) Unit. The Unit is supported by the Canadian Institutes of Health Research (CIHR) and provincial partners including the Ministère de la Santé et des Services sociaux (MSSS) du Québec and the Fonds de recherche du Québec – Santé (FRQ-S). The funders have no role in developing the protocol.
Ethics approval and consent to participate
For this project, the ethical approval is not required because we will only use the literature as the data source. Also, there is no requirements of an inform consent.
Consent for publication
Our manuscript contains no individual person’s data in any form including any individual details, images, or videos.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Zomahoun, H.T.V., Massougbodji, J., Bussières, A. et al. Improving the usefulness of evidence concerning the effectiveness of implementation strategies for knowledge products in primary healthcare: protocol for a series of systematic reviews. Syst Rev 9, 112 (2020). https://doi.org/10.1186/s13643-020-01382-x
- Implementation strategies
- Primary healthcare
- Systematic reviews