Areas of work of the advisory group
Area 1 Advancing the science of HSR synthesis by developing and strengthening networks between individuals and institutions and collaborations and groups that have an interest in HSR synthesis and translation
The Advisory Group has evolved to include participants from 17 organizations/networks who convene once every 3 to 4 months to discuss current challenges and advances related to both producing systematic reviews and drawing on them for guidance on strengthening health systems. The group shares information, identifies where further methodological work or training is needed and builds cross-institutional links, and thereby, with its anchoring at the Alliance and WHO, serves as a global focal point for coordination around the issues.
Area 2 Providing support, information sharing and coordination related to setting priorities for HSR synthesis regionally and globally and increasing capacity building in HSR synthesis and translation (particularly in relation to LMICs)
The Advisory Group, together with the Alliance more broadly, has contributed to setting priorities for health systems research on a global scale [17, 18]. The systematic review centres funded by the Alliance have developed priority-setting approaches for HSR syntheses and, in doing so, have implemented different mechanisms to engage with decision-makers, to increase the likelihood of the synthesised evidence being applied in practice. For instance, the South African Medical Research Council developed an electronic survey addressed to both academics and practitioners/policymakers, in order to identify key HSR issues to be reviewed. Results of the survey informed a preselection phase in which existing systematic reviews relevant to the HSR issues identified were summarised. This process informed the ranking of a set of the highest priority questions by a steering committee and advisory group, which included policymakers at national and regional levels. Another example is the Center for Systematic Reviews of Health Policy and Systems Research (SPARK) at the American University of Beirut in Lebanon, which is developing a tool for prioritizing questions for HSR systematic reviews. Although priority-setting approaches exist for some types of research [19, 20], little work addresses how to engage policymakers and other stakeholders in prioritizing research [21] and there is no published framework to guide priority setting for health policy and systems research synthesis.
Limited capacity and issues of financial sustainability of groups in LMICs that undertake systematic reviews are further important challenges identified by the Advisory Group. For example, how can these groups secure core funding, ensure the availability of specialised resources such as information scientist support and statistical tools, strengthen their capacity to undertake the wide range of reviews needed to inform health system decisions and improve their engagement with decision-makers in the review process? Discussions on these issues contributed to the development of the new Global Evidence Synthesis Initiative (GESI), led by an international collaborating group of organisationsFootnote 1 and convened by Cochrane. It is anticipated that GESI will coordinate and support the development of capacities for conducting systematic reviews (not necessarily limited to health systems evidence or even the health sector) in LMICs, building on the resources and networks of the collaborating organisations.
Another important challenge identified by the Advisory Group relates to strengthening the capacity of decision-makers and stakeholders to access, assess and apply HSR evidence in decision-making and to create a “user-pull” for HSR evidence. The Advisory Group advocates for and supports the development of user-friendly summaries of HSR syntheses and recognises the importance of sensitization workshops for users of HSR syntheses, institutionalised mechanisms for researcher-user interactions and knowledge translation activities. The last involves using knowledge translation tools such as evidence briefs for policy and briefing notes and convening dialogue sessions with decision-makers and stakeholders.
Area 3 Expanding the range of study designs that can be included in reviews of the effectiveness of HSR interventions
For a number of areas within the field of HSR, evidence from quasi-experimental studies forms a substantive component of the overall evidence base for questions on the impacts of interventions. Evidence from quasi-experimental studies may be particularly important for reviews of interventions or exposures whose effects are not, often for practical reasons, easily amenable to measurement using designs based on random assignment of participants or facilities to comparison groups under the direct control of researchers. Quasi-experimental studies, such as regression discontinuity designs, [22, 23] interrupted time series [24] and instrumental variable analyses [25], offer potential for inferences to be made about the causal effects of health systems interventions and reforms. These inferences can be as valid as those derived from randomised controlled trials—without intervening externally in the health system. Quasi-experimental studies thus allow the study of intervention and reform effectiveness in their natural setting.
However, efforts to integrate quasi-experimental studies into effectiveness review frameworks in both HICs and LMICs are, in general, at an early stage of development [26–28]. This is in part because methods and standards for incorporating evidence from quasi-experimental studies are underdeveloped for most fields of study, including HSR [29]. Review authors therefore do not always include these study designs when it might be appropriate to do so. For instance, two recent studies of systematic reviews of health systems evidence found wide variation in their inclusion of quasi-experimental studies and suggested that more work is needed on when it may be useful to do so [29, 30].
With the Harvard School of Public Health, the Alliance organized a workshop in 2013 focused on the following: (1) establishing a taxonomy of quasi-experimental study designs, (2) developing guidelines for systematic review authors on incorporating quasi-experimental studies in reviews, (3) devising strategies to build institutional capacity to support the wider inclusion of quasi-experimental studies in reviews, (4) establishing approaches to translate evidence from quasi-experimental studies for policymakers, and (5) strengthening partnerships between researchers in HICs and researchers in LMICs.
One of the outputs of the workshop and the background work commissioned by the Alliance is the identification of five quasi-experimental study designs frequently employed in health systems research: natural experiments, instrumental variable analyses, regression discontinuity analyses, interrupted times series studies and difference studies including controlled before-and-after designs, difference-in-difference designs and fixed effects analyses of panel data [31].
Area 4 Piloting a system for producing policy relevant syntheses/systematic reviews of HSR addressing questions other than effectiveness
Systematic reviews addressing the full range of health system issues may be useful for decision-making (Table 1). However, guidance around syntheses for questions other than effectiveness of interventions remains limited. Further guidance is needed on templates for reporting results, tools to assess confidence in review results and institutional mechanisms to facilitate evidence uptake.
The EPPI Centre is leading a study, funded by the Alliance, on supporting policy-relevant systematic reviews for health systems. Interviews with policymakers and systematic reviewers have emphasised the importance of connecting these two groups and supporting their mutual engagement with formalised procedures and structures, so as to ensure that their motivations for producing and using systematic reviews are aligned in terms of the urgency and generalizability of products [32].
In addition, members of the Advisory Group are conducting scoping reviews to assess the most appropriate knowledge synthesis methods to answer different types of research questions [33, 34]. These reviews aim to identify, define and classify emerging knowledge synthesis methods.
Area 5 Advocating for and supporting a common global database for all types of systematic reviews of HSR
The Advisory Group acknowledges the utility and promotes the use of repositories of evidence synthesis in this field, such as Health Systems Evidence (HSE) (www.healthsystemsevidence.org), a database for all types of systematic reviews of HSR. In this database, systematic reviews, and links to their included primary studies, are complemented by a range of other policy-relevant documents addressing health systems, including evidence briefs for policymakers, overviews of systematic reviews, systematic review protocols, economic evaluations and descriptions of health system reforms. The database is available in Arabic, Chinese, English, French, Portuguese, Russian and Spanish; it can be searched using country, region and LMIC filters; it includes links to user-friendly summaries written by other groups. These features make the database a particularly powerful resource for health system decision-makers, stakeholders and researchers. HSE also now includes the beta version for an Intergovernmental Organizations’ Health Systems Documents Portal, developed as a complete inventory of all policy-relevant WHO documents about health systems. Another useful repository is PDQ-Evidence (http://www.pdq-evidence.org), which links systematic reviews of HSR and other health research, overviews of reviews and their included primary studies.
These databases are useful in identifying existing reviews that could inform decisions on specific HSR issues and should also be consulted at the planning stages for new primary health system research and for reviews. This would allow identification of gaps in the HSR literature and key areas for further HSR, including implementation research (Table 1).
Area 6 Advocating for and supporting prospective registration of all protocols for systematic reviews when they are planned
Prospective registration of systematic reviews is an important step in reducing duplication of effort, fostering collaboration on conducting reviews and enabling comparison of the completed review with what was planned originally. International prospective register of systematic reviews (PROSPERO), the international prospective register of systematic reviews (http://www.crd.york.ac.uk/prospero), was established for this purpose but initially included only reviews of effectiveness. The Advisory Group therefore initiated discussions with PROSPERO regarding how the full range of health systems systematic reviews (Table 1) could be accommodated. This feedback, as well as findings from a survey of users [35], led to modifications to provide a system sufficiently flexible to accommodate a range of review types.