Priority domains, aims, and testable hypotheses for implementation research: A scoping review and evidence map protocol

Background: The challenge of implementing evidence-based innovations within practice settings is a significant public health issue the field of implementation research (IR) is focused on addressing. Significant amounts of funding, time, and effort have been invested in IR to date, yet there remains significant room for advancement, especially regarding IR’s development of scientific theories as defined by the National Academy of Sciences (i.e., a comprehensive explanation of the relationship between variables that is supported by a vast body of evidence). Research priority setting (i.e., promoting consensus about areas where research effort will have wide benefits to society) is a key approach to helping accelerate research advancements. Thus, building upon existing IR, general principles of data reduction, and a general framework for moderated mediation, this article identifies priority domains, aims, and testable hypotheses for IR and describes a scoping review protocol to identify and map the extent to which IR has examined these priorities to date. Methods: Implementation Science is the leading journal for publishing IR and receives over 800 submissions annually. Thus, this scoping review will focus on IR published in Implementation Science between its inception in 2006 and 12/31/2019. The current scoping review and evidence map protocol has been developed in accordance with the approach developed by Arksey & O’Malley and advanced by Levac, Colquhoun, and O’Brien. All research articles and short reports will be reviewed. Because scoping reviews seek to provide an overview of the identified evidence base rather than synthesize findings from across studies, we plan to use our data-charting form to provide a descriptive overview of implementation research to-date and summarize the research via one or more summary tables. We will use the priority aims and testable hypotheses (PATH) diagram, which integrates the four priority domains, three priority aims, and four priority testable hypotheses, to develop a map of the evidence (or lack thereof). Discussion: This scoping review and evidence map is intended to help accelerate IR focused on one or more of IR’s priority aims and testable hypotheses, which in turn will accelerate IR’s development of NAS-defined scientific theories and, subsequently, improvements in public health. Systematic review registration: Open Science Framework:

we have advanced them as consistent, valid, and efficient measures of implementations. 21 Otherwise, we rely on the assumption that implementation outcomes are predictive of HHROs, without empirically demonstrating this to be true. The PATH4IR Project is the first to explicitly identify EBMIs as a priority domain for IR.

Context-related moderators/mediators. Moderation occurs when the effect of an independent
measure on a dependent variable depends on the level of another measure and mediation occurs when the effect of an independent variable on a dependent measure is transmitted through a third variable. 28 Given that existing IR models/frameworks have highlighted the importance of context 19,20,29 and that Edwards and Lambert's (2007) general framework for moderated mediation 24 guided identification of the priority domains for this project, Context-Related Moderators/Mediators (CRMM) was identified as a priority domain for IR. Including CRMM as a priority domain is consistent with several IR models/ frameworks. 19,20,29 IR models/frameworks that do not include a contextrelated domain 22,30 are limited given the hypothesized importance of inner and outer contextual factors in EBP implementation.

Three priority aims for implementation research
There are numerous aims (i.e., research questions) that IR could address, but not all aims have equal significance. Relative to IR's domains, IR's aims have received less explicit attention. The work of Curran et al. (2012) 31 is one exception. Specifically, for their type 3 effectiveness-implementation research categorization, Curran et al. recommended that the primary aim of this research category was to "determine utility of an implementation intervention/strategy" and the secondary aim was to "assess clinical outcomes associated with implementation trial." 31 Curran et al. also recommended implementation outcomes (i.e., adoption, fidelity) as dependent measures for the primary aim, with client outcomes (e.g., patient symptoms patient functioning) as dependent measures for the secondary aim. 31 However, priority aims have not generally been explicitly addressed by most other IR models/frameworks. 19,20,22 Given that developing or contributing to generalizable knowledge is central to how research is defined, 32 it is important that IR prioritize aims that seek to develop or contribute to generalizable knowledge for its priority relationships. Thus, building from the four priority domains described above, we identified the following three priority aims for IR: (1) the IS to HHRO relationship (i.e., IS → HHRO), (2) the IS to EBMI relationship (i.e., IS → EBMI), and (3) the EBMI to HRRO relationship (i.e., EBMI → HRRO). Drawing from mediational analysis literature, [33][34][35][36] we have termed IR focused on the IS → HHRO relationship as Path C IR (the red triangle of Figure 1), IR focused on the IS → EBMI as Path A IR (the blue triangle of Figure 1), and IR focused on the EBMI → HHRO relationship as Path B IR (the green triangle of Figure 1). Each priority aim is defined below and in Table 3.

Advance generalizable knowledge regarding the IS → HHRO relationship.
Advancing generalizable knowledge about the relationship between an IS and a HHRO is termed Path C IR. Given IR's emphasis on strategies to increase the uptake of EBPs to improve patient and population health, [4][5][6] Path C IR was identified as a priority aim for IR. However, further support for Path C IR as a priority aim is provided by Foy et al., who as noted above, suggested "If studies evaluating the effects of implementation interventions are to be of relevance to policy and practice, they should have endpoints related to evidence-based processes of care." 37 An example of Path C IR is a 29-site cluster randomized implementation experiment Garner et al. (2012) conducted between 2008 and 2012 that focused on testing the impact of a pay-for-performance (P4P) IS to improve the implementation and effectiveness of the Adolescent Community Reinforcement Approach (A-CRA), which is an EBP for adolescents with substance use disorders. 15 For the primary HHRO, which was adolescent substance use recovery status at 6-month follow-up, no significant difference between IS conditions was found. 15

Advance generalizable knowledge regarding the IS → EBMI relationship.
Advancing generalizable knowledge about the relationship between an IS and an EBMI is termed Path A IR. Given that an EBMI is a measure of EBP implementation found to be predictive of key client outcomes (i.e., HHROs), 9 Path A IR was identified as a priority aim for IR. Relative to IR that has tested the impact of an IS on implementation outcomes that do not have evidence of being a meaningful predictor of key client outcomes, IR testing the impact of an IS on EBMIs appears be limited. However, having established an EBMI for A-CRA as part of an effectiveness study, 9,14 Garner et al. (2012) 15 also provide an example of Path A IR. Indeed, examining the impact of P4P on an EBMI called Target A-CRA (i.e., 10+ of the core A-CRA components delivered within no less than seven sessions), which prior research found to be significantly associated with greater reductions in adolescents' days of abstinence at follow-up, 14 Garner et al. (2012) found that relative to adolescents in the implementation-as-usual (IAU) condition, adolescents in the P4P condition had a significantly higher likelihood of receiving Target A-CRA. 15

Advance generalizable knowledge regarding the EBMI → HHRO relationship.
Advancing generalizable knowledge about the relationship between an EBMI and HHROs is termed Path B IR.
Research by Nosek et al. (2015), 38 which increased concern regarding the reproducibility of psychological science, underscores why Path B IR is a priority. That is, it is important that significant relationships (e.g., EBMI → HHRO) supported as part of effectiveness research be examined for replicability within implementation research. As part of their IR experiment, Garner et al. (2012) provide an example of Path B IR by replicating a significant association between Target A-CRA (i.e., the EBMI) and adolescent abstinence from substance use at follow-up (i.e., the HHRO). 15

Four priority testable hypotheses for implementation research
The possible testable hypotheses for IR are numerous. However, because not all hypotheses are equally significant, there is value in establishing consensus regarding the types of testable hypotheses IR should prioritize. Toward helping generate NAS-defined scientific IR, prioritizing one or more of the four testable hypotheses shown in  Table 4.

Effectiveness hypotheses from a superiority trial.
Testing the extent to which an experimental IS has superior effectiveness, relative to an active-control IS, is termed IR testing an upper left quadrant (ULQ) hypothesis. In contrast to research on clinical treatments, where an active-control condition may not exist or be appropriate, IR should include the most appropriate active-control IS possible. One of the most appropriate active-control condition IS may be the IS used as part of an EBPs effectiveness research. To date, the "large and growing evidence base relating to the effectiveness of implementation strategies" noted by Foy et al. 39 has tested ULQ hypotheses and supports that this testable hypothesis is and should remain a priority for IR. Indeed, given that tests of ULQ hypotheses may continue to be the most common type of IR hypotheses, it may not be much longer before results of ULQ hypothesis tests are analyzed as part of a meta-analysis.

Cost-effectiveness hypotheses from a superiority trial.
Testing the cost-effectiveness of an IS that has been shown to have superior effectiveness, relative to an active-control IS, is termed IR testing an upper right quadrant (URQ) hypothesis. It is considered a priority testable hypothesis for IR as knowing the effectiveness of an intervention/strategy is not sufficient for many potential users, especially decision makers who need to know whether the benefits from the intervention/strategy are commensurate with its costs (i.e., whether it delivers value), 40 18 provide an example of IR testing a URQ hypothesis. Supporting the cost-effectiveness of a P4P IS, Garner et al (2018) found that although the P4P strategy led to significantly higher average total costs compared to the IAU control condition, this average increase of 5% resulted in a 325% increase in the average number of patients who received Target A-CRA (i.e., the EBMI). 18 Effectiveness hypotheses from non-inferiority trial. Testing the extent to which an experimental IS has non-inferior effectiveness, relative to an active-control IS, is termed IR testing a lower left quadrant (LLQ) hypothesis. Similar to how Schumi and Wittes (2011) 41 explain noninferiority, testing a non-inferiority hypothesis seeks to provide evidence that the IS being tested is "not unacceptably worse" than the IS being used as a control. This is a priority for IR given strategies used to study a clinical intervention's effectiveness may not be possible in practice settings (e.g., too intensive). We are not aware of IR that has tested LLQ hypotheses. However, a close example is a non-randomized observational IR study by Stirman et al. (2017) 42

Cost-effectiveness hypotheses from non-inferiority trial. Testing the cost-effectiveness of an IS
shown to have non-inferior effectiveness, relative to an active-control IS, is termed IR testing a lower right quadrant (LRQ) hypothesis. Again, given decision makers desire to know the extent to which benefits from an IS are commensurate with its costs, 40 44 to support the non-inferiority of injectable hydromorphone hydrochloride (i.e., a narcotic pain reliever) relative to injectable diacetylmorphine hydrochloride (i.e., pharmaceutical heroin).

Objectives
The primary objective of the PATH4IR Project's scoping review is to advance understanding regarding the extent to which IR to date has (or has not) examined the four priority domains, three priority aims, and four priority testable hypotheses described above. We hypothesize that IR addressing these priorities will be limited (i.e., represent significant gaps in the extant IR literature). Thus, a secondary objective of this review is to help advance understanding regarding what domains, aims, and testable hypotheses IR has focused on to date.

Stage 1: Identifying the research questions
The primary research questions our research team will answer with this scoping review is: To what extent have the four priority domains, three primary aims, and four priority testable hypotheses described above been addressed by IR to date? Via an iterative process, our research team also identified the following secondary research questions: (1) which other domains have been studied by IR to date, (2) which other aims have been studied by IR to date, and (3) which other hypotheses have been examined by IR to date.

Stage 2: Identifying relevant studies
Implementation Science is the leading journal for publishing IR and receives over 800 submissions annually. 47 As such, this review will focus on IR published in Implementation Science since its inception in 2006. To identify relevant studies, we will search PubMed using the search strategy below and cross-reference the results with a list of publications on the journal's website:

Search "Implementation science IS"[Journal]
Filters: Publication date to 12/31/2019 "Research articles" and "short reports" published in Implementation Science since its inception through 2019 are eligible. Articles labeled by the journal as "systematic review," "methodology," debate," or "conference proceedings" are not eligible as this review aims to map original IR.
"Protocols" were also excluded given that intended analyses do not always align with published results. Research articles and short reports will be excluded if the review team agrees that the paper's primary objective is more aligned with an excluded article type.

Stage 3: Study selection
Reference information and full texts for all articles published in Implementation Science in 2019 or earlier will be imported into an EndNote database. The articles will be sorted by a reviewer by type to identify all articles labeled by the journal as research articles or short reports. In the subsequent stages, if a reviewer encounters an article deemed ineligible (i.e., labeled by the journal as a research article or short report but is not considered primary implementation research), the reviewer will raise it with review team so that consensus around an inclusion decision can be reached. Table 5 provides a list of variables to be included in the project's data-charting form, which was developed based on discussions by the review team regarding what information should be recorded for each eligible article and a pilot test of the form with five articles. First author, title, publication

Stage 4: Charting the data
year, and article type are included as article identifiers. We will extract whether and which IS, HHRO, EBMI, or CRMM was studied, which relationships between these domains were studied (i.e., Path C, A, or B), and whether URQ, ULQ, LLQ, or LRQ hypotheses were tested when studying these relationships to answer our primary question of the extent to which the priority domains, aims, and testable hypotheses have been assessed in IR to date. As a secondary question, we will seek to understand what other domains, aims, and testable hypotheses have been examined by IR to date. For example, we will extract whether studies consider implementation outcomes that are not evidence-based or contextual factors not as moderators or mediators to understand which other domains have been examined and the extent to which they have been examined. We anticipate identifying IR that focused on implementation outcomes rather than EBMIs and therefore will record whether the IS → implementation outcome relationship was assessed. Our form also will include a space to capture other aims and testable hypotheses that IR has examined to date.
To ensure validity of the form, data will be extracted by a primary reviewer and confirmed by a secondary reviewer for approximately one-third of the included articles. Any conflicts will be discussed until consensus is reached. Clarifications and additional revisions to the data-charting form based on the types of conflicts that arise will be considered. Once the form is finalized at this stage, data from the remaining articles will be extracted by a single reviewer.

Stage 5: Collating, summarizing, and reporting the results
A PRISMA flow diagram will be used to report results of the scoping review. Additionally, we will present a descriptive overview (including tabular and/or graphical summaries) of the eligible full texts. Because scoping reviews seek to provide an overview of the identified evidence base rather than synthesize findings from across studies, we plan to use our data-charting form to provide a descriptive overview of implementation research to-date and summarize the research via one or more summary tables (e.g., for each priority aim). Additionally, we will use the priority aims and testable hypotheses (PATH) diagram (see Figure 3), which integrates the four priority domains, three priority aims, and four priority testable hypotheses, to develop a map of the evidence (or lack thereof).

Discussion
Despite significant amounts of funding, time, and effort, the field of IR has yet to develop scientific theories as defined by the NAS (i.e., a comprehensive explanation of some aspect of nature that is supported by a vast body of evidence). The findings from this project are intended to help accelerate IR focused on one or more of the identified IR priority aims and testable hypotheses, which in turn will accelerate IR's development of NAS-defined scientific theories and, subsequently, improvements in public health. Results of this scoping review will be disseminated via presentations at professional conferences (e.g., Annual Conference on the Science of Dissemination and Implementation in Health, Society on Implementation Research Collaboration), publication in in a peer-reviewed journal (e.g.,

Implementation Science, Implementation Research and Practice, Implementation Science
Communications).       Priority hypotheses for implementation research The PATH diagram for implementation research

Supplementary Files
This is a list of supplementary files associated with this preprint. Click to download.