Types of studies
We will include randomized controlled trials (RCTs), non-randomized controlled trials (NRCTs), controlled before-after studies (CBA), and interrupted time series (ITS) studies.
Types of participants
We will include studies which include health policy-makers at all levels (including civil society organization staff, non-governmental organization staff, local government staff, federal government staff) and health system managers making decisions on behalf of a large jurisdiction or organization (6). We will not include studies related to decision-making for an individual person or patient.
Types of interventions
We will include studies examining any type of “friendly front end,” “evidence summary,” or “policy brief” or other product derived from systematic reviews or guidelines based on systematic reviews that presents evidence in a summarized form to policy-makers and health system managers. Interventions must include a summary of a systematic review and be actively “pushed” to target users. For example, a potentially included study used an intervention that evaluated the effectiveness of friendly front ends by assessing changes in policy-maker beliefs [12]. An example of a study that would be excluded assessed the views of policymakers on how systematic reviews can be promoted within a low- and middle-income country [13].
We will exclude studies in which evidence summaries are one component of a multi-component intervention.
Comparison
We will include any comparisons including active comparators (e.g., other summary formats) or no intervention.
Types of outcome measures
Primary Outcomes
The primary outcomes are the following:
-
1.
Use of systematic review derivative product in decision-making (e.g., self-reported use of the evidence in policy-making and decision-making as well as self-reported access of research, appraisal of research, or commissioning of further research within the decision-making process [14]. We will include instrumental use of research in decision-making (e.g., direct use of research) as well as conceptual use (e.g., using research to gain an understanding of a problem or intervention) and symbolic use (e.g., using research to confirm a policy/program already implemented) [15].
-
2.
Understanding, knowledge, and/or beliefs (e.g., changes in knowledge scores about the topic included in the summary).
Secondary Outcomes
We will also include studies that report on any of the following outcomes:
-
Perceived relevance of systematic review summaries
-
Perceived credibility of the summaries
-
Perceived usefulness and usability of systematic review summaries
-
Understandability of summaries
-
Desirability of summaries (e.g., layout, selection of images, etc) [4]
We recognize that some studies may use different terms to describe these outcomes. For example, the term “satisfaction” maybe used as an umbrella term to capture relevance, usability, and desirability. These outcomes will be assessed by the team and categorized according to the above list.
This systematic review has not been registered with PROSPERO since there we are not assessing health outcomes.
Search methods for identification of studies
An information specialist will help develop the search strategy using the PRESS Guideline [16]. We will build on the search strategy used by Perrier et al. and Murthy et al. in their systematic reviews of interventions to encourage the use of systematic reviews by health managers and policy-makers [9, 10].
Electronic searches
The search conducted by Perrier et al. identified 11,297 records (after removing duplicates) and included four papers reporting two studies. We will expand this search by including additional databases, as suggested by John Eyres, of the International Initiative for Impact Evaluation (3ie) and the Campbell International Development Review Group. These include databases such as Global Health (CABI), Global Health Library (from WHO), Popline, Africa-wide, Public Affairs Information Service, Worldwide Political Science Abstracts, Web of Science, and DfiD (Research for Development Database) (see Additional file 1).
Searching other resources
We will search websites of research groups and organizations producing evidence summaries to identify unpublished studies evaluating the effectiveness of the systematic review derivatives in increasing policy-makers’ understanding (e.g., Health Systems Evidence, the Canadian Agencies for Health Technology Assessment, SUPPORT Summaries).
We will check reference lists of relevant studies to identify additional studies. We will contact researchers to identify ongoing and completed/published work. We will report the results of the search using the PRISMA flow diagram.
Data collection and analysis
Selection of studies
Two reviewers will independently screen titles and abstracts to identify relevant studies meeting the pre-specified inclusion criteria. The full text of potentially included studies will be screened independently by two authors. Data extraction and quality assessment will be conducted independently and in duplicate. We will use software Covidence (https://www.covidence.org/) for screening of studies. All completed studies will be included if they meet the inclusion criteria listed above.
Data extraction and management
The data extraction form will be pre-tested and will include factors related to the population, intervention, comparison, and outcomes. The data will be extracted independently in duplicate by two reviewers using a structured Excel sheet and will be piloted on ten articles. Disagreements on extractions will be resolved by discussion and with a third member of the research team when necessary Evidence summaries, like systematic reviews, that seek to inform decisions in a neutral way should not contain recommendations. Therefore, summaries that provide recommendations will be assessed separately from those without recommendations since these may affect the user experience [17].
Data will be extracted for the following:
-
Country
-
Setting
-
Study design
-
Participants
-
Intervention
-
Type of evidence summary
-
Format of evidence summary
-
Description of evidence summary components (e.g., descriptions of easy-to-skim formatting, graded entry, use of tables/figures) [18]
-
Mode of delivery
-
Topic of evidence summary
-
Recommendation of evidence summary
-
Outcomes
-
Policy/decision-makers’ self-reported use of summaries in decision-making
-
Policy/decision-makers’ knowledge of the summary content and the measurement used
-
Policy/decision-makers’ understanding and measurement used
-
Perceived relevance of the summaries and measurement used
-
Perceived credibility of the summaries and measurement used
-
Perceived usefulness and usability of the summaries and measurement used
-
Perceived understandability of the summaries and measurement used
-
Perceived desirability of the summaries and measurement used
-
Process indicators
-
How the systematic review was selected for summary (e.g., based on topic, quality criteria)
-
How the evidence summary was developed (e.g., iterative process)
-
Involvement of stakeholders in evidence summary development—which stakeholders, description of involvement
Assessment of risk of bias in included studies
The methodological quality will be specifically examined using the risk of bias tools from the Cochrane Handbook for randomized trials and the Effective Practice and Organization of Care (EPOC) Review Group criteria for interrupted time series and controlled before-after studies [19, 20] and A Cochrane Risk Of Bias Assessment Tool: for Non-Randomized Studies of Interventions (ACROBAT-NRSI) [21].
Measures of treatment effect
Effect estimates and confidence intervals for individual studies will be calculated (where possible) irrespective of whether a pooled effect estimate is calculated. When it is possible to combine studies, dichotomous outcomes will be reported as relative risks. Continuous outcomes will be reported as weighted mean differences. If an outcome has been reported in different scales (e.g., understanding), and we consider the scales to measure a similar construct, standardized mean differences will be used to summarize the data. When it is not possible to combine the data, we will present the results for each study separately.
Unit of analysis issues
When possible, any studies with cluster allocation (e.g., cluster-randomized trials, cluster-allocated controlled before and after studies, and interrupted time series) analyses with errors in the unit of analysis will be adjusted using the variance inflation factor, as described in the Cochrane Handbook, if the necessary data can be obtained from the study authors. We will obtain ICC from other similar studies with similar outcomes if the ICC is not published (e.g., by checking the Aberdeen website of ICCs, http://www.abdn.ac.uk/hsru/research/delivery/behaviour/methodological-research/ or the Campbell Collaboration website of ICCs for education). Sensitivity analyses will be used to assess the effects of incorporating these corrected analyses in our analysis.
Dealing with missing data
We will attempt to contact the contact author of the studies by email for any missing data.
Assessment of heterogeneity
If meta-analysis is possible, we will explore heterogeneity using forest plots and the I2 statistic according to guidance of the Cochrane Handbook for Systematic Reviews of Interventions [19].
Assessment of reporting biases
If more than ten studies are included, we will use funnel plots to explore publication bias.
Data synthesis
Where appropriate, results will be synthesized using meta-analysis. We will present the relative risks using random effects models for dichotomous outcomes and standardized mean differences for continuous outcomes. When studies have reported the same outcome using different scales, we will use standardized mean differences. Non-randomized studies will be meta-analyzed separately from RCTs. When results cannot be pooled, we will present a narrative summary of the results.
We will analyze the results of qualitative data from included studies, when possible, to understand the perceptions and attitudes regarding the components of the summaries that were considered the most useful.
Assessing the methodological quality
We will use the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to assess the quality of evidence for the outcomes reported in this review [22].
Subgroup analysis and investigation of heterogeneity
Heterogeneity will be explored, if possible, by conducting meta-regression to assess the role of mediating factors, including the following:
-
Target audience of summary (e.g., focused on specific local context, generic summary)
-
Type of decision maker (e.g., federal policy-maker versus hospital administrator)
-
Components of friendly front end (e.g., bulleted list, text, summary of findings table, causal chain)
Sensitivity analysis
The impact of including studies assessed as high risk of bias or studies in which there were unit of analysis errors that could not be reanalyzed will be considered in sensitivity analysis.
Applicability
We will assess applicability of the findings of the review to specific settings of relevance to end-users. We will use the most up to date methods from the Cochrane Applicability and Recommendations Methods Group which include assessing the directness of the evidence to specified settings of interest using GRADE [23].
