- Open Access
- Open Peer Review
Stakeholder involvement in systematic reviews: a scoping review
Systematic Reviewsvolume 7, Article number: 208 (2018)
There is increasing recognition that it is good practice to involve stakeholders (meaning patients, the public, health professionals and others) in systematic reviews, but limited evidence about how best to do this. We aimed to document the evidence-base relating to stakeholder involvement in systematic reviews and to use this evidence to describe how stakeholders have been involved in systematic reviews.
We carried out a scoping review, following a published protocol. We searched multiple electronic databases (2010–2016), using a stepwise searching approach, supplemented with hand searching. Two authors independently screened and discussed the first 500 abstracts and, after clarifying selection criteria, screened a further 500. Agreement on screening decisions was 97%, so screening was done by one reviewer only. Pre-planned data extraction was completed, and the comprehensiveness of the description of methods of involvement judged. Additional data extraction was completed for papers judged to have most comprehensive descriptions. Three stakeholder representatives were co-authors for this systematic review.
We included 291 papers in which stakeholders were involved in a systematic review. Thirty percent involved patients and/or carers. Thirty-two percent were from the USA, 26% from the UK and 10% from Canada. Ten percent (32 reviews) were judged to provide a comprehensive description of methods of involving stakeholders. Sixty-nine percent (22/32) personally invited people to be involved; 22% (7/32) advertised opportunities to the general population. Eighty-one percent (26/32) had between 1 and 20 face-to-face meetings, with 83% of these holding ≤ 4 meetings. Meetings lasted 1 h to ½ day. Nineteen percent (6/32) used a Delphi method, most often involving three electronic rounds. Details of ethical approval were reported by 10/32. Expenses were reported to be paid to people involved in 8/32 systematic reviews.
We identified a relatively large number (291) of papers reporting stakeholder involvement in systematic reviews, but the quality of reporting was generally very poor. Information from a subset of papers judged to provide the best descriptions of stakeholder involvement in systematic reviews provide examples of different ways in which stakeholders have been involved in systematic reviews. These examples arguably currently provide the best available information to inform and guide decisions around the planning of stakeholder involvement within future systematic reviews. This evidence has been used to develop online learning resources.
Systematic review registration
The protocol for this systematic review was published on 21 April 2017. Publication reference: Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, Watts C, Morley R: Stakeholder involvement in systematic reviews: a protocol for a systematic review of methods, outcomes and effects. Research Involvement and Engagement 2017, 3:9. https://doi.org/10.1186/s40900-017-0060-4.
The concept of active involvement in research of people with a healthcare condition, their families, friends and carers, was founded on the principle that people affected by the condition have a moral right to contribute to decisions about what research is undertaken and in what way [1,2,3]. The active involvement of other stakeholders (meaning patients, the public, health professionals, health decision makers and funders) grew from a desire to address the lack of real-world relevance of research and to ensure more effective implementation of research findings into practice [4, 5]. It is now widely accepted in many parts of the world that the active involvement of many of these groups (that we collectively refer to as ‘stakeholders’) is beneficial to the quality, relevance and impact of health research [2, 3]. Accordingly, many funding bodies, including government and charities, now mandate that researchers actively involve patients and the public in their research, including systematic reviews [6,7,8,9], although there is evidence of international variation in the extent to which patients and the public are involved .
Systematic reviews aim to inform and support the delivery of evidence-based practice, by finding and bringing together, in an explicit and transparent way, all the research evidence that addresses a particular topic or healthcare question. Stakeholder involvement within systematic reviews has been proposed as a way to enhance the actual and perceived usefulness of synthesised research evidence, addressing barriers to the uptake of evidence into practice . In this paper, we define (based on a number of published definitions, e.g. [1, 12, 13]) ‘active stakeholder involvement’ as the contribution of people who are not researchers throughout the process of production and dissemination of a systemic review, including the planning and conduct of an individual systematic review. While there are a number of examples of active stakeholder involvement in systematic reviews, the approaches to, and extent of, involvement have varied considerably [14,15,16] and synthesised evidence and resources to guide practice is lacking. As well as active involvement within individual systematic reviews, stakeholders may also get involved at the level of organisations which commission or carry out systematic reviews. A recent review explored examples of consumer involvement within organisations (such as Cochrane) that support production of systematic reviews , but evidence relating to relevant activities and roles of individual researchers and how they may involve stakeholders in their reviews remains scant .
As part of a wider project to provide guidance to researchers about how to involve stakeholders in systematic reviews , we undertook a mixed-method evidence synthesis, first completing a scoping review to create a broad map of evidence relating to stakeholder involvement in systematic reviews, followed by two contingent syntheses . Here, we report the results of the scoping review. The aims of this paper are therefore to:
Document the evidence-base relating to stakeholder involvement in systematic reviews
Use this evidence to describe key features of how stakeholders have been involved in systematic reviews
We carried out a scoping review, following a protocol . We followed the methodological steps outlined by Arksey and O’Malley  and used an iterative team approach, with regular team meetings to discuss progress and reach consensus on next steps, to ensure clarity of purpose and balance between breadth and comprehensiveness of the review [21,22,23]. Protocol deviations, with justifications, are described in Additional file 1.
We implemented a stepwise approach  to promote efficient identification of up-to-date literature, balancing the expected large volume of literature with available time and resources. Details of this approach, including pre-agreed criteria and contingencies to inform decisions relating to the extent of the searches, have previously been described ; below we report the actual steps of searching and brief justification for these steps.
We used a comprehensive search strategy, adapted for each database (see Additional file 2). In step 1, we searched a comprehensive set of databases (CENTRAL (CDSR, DARE, HTA, Cochrane Methodology Register), Embase (Ovid), MEDLINE (Ovid), CINAHL (EBSCO), AMED, Joanna Briggs Database and ProQuest Dissertations and Theses (handsearched)), within a narrow time period (from 01 January 2014 to 09 April 2016). The aim of step 1 was to, in an efficient way, identify the databases most likely to include relevant papers. In step 2, we searched a more limited set of databases (Embase, MEDLINE, CINAHL and HTA) for a longer time period (01 January 2010 to 31 December 2013) with the aim of exploring whether there was justification for extending the search beyond 2010.
Searching and application of inclusion criteria was applied to each step prior to progression to the next step. For step 1, we noted the source database (or databases) of each identified record, and the databases from which the greatest number of included papers were identified. The results of these explorations were discussed and review team consensus reached on which databases to include in step 2. After step 2, the review team explored the publication dates of records meeting our inclusion criteria. The majority of papers meeting our inclusion criteria (63%) were published in either 2014 or 2015 (see Additional file 3). The sharp drop in numbers of included papers from 2014 to 2013 and relatively stable number of included papers between 2013 and 2010 were key factors in the team decision not to extend electronic searching to before 2010.
Additional sources we searched include the reference lists of recent relevant reports and reviews (e.g. [6, 17, 25]), the reference lists of all included studies and articles published in the journal Research Involvement and Engagement. To identify unpublished reports, we contacted authors of published papers and promoted this review via social media.
Selection criteria for inclusion were purposefully wide. We included any paper, published or unpublished, regardless of study design, including commentaries, letters and expert opinion, which investigated, reported or discussed any aspect of stakeholder involvement in a systematic review. We anticipated that we would include (but would not be limited to) evidence such as published systematic reviews which reported involvement; reports of methods of involvement in an individual systematic review; studies quantitatively or qualitatively evaluating involvement in individual systematic reviews; and opinions, commentary and discussion relating to involvement in systematic reviews.
We excluded papers that focussed on stakeholder involvement in the generation of research priorities (unless they were specifically generating questions for a systematic review) and in both research more broadly, and guideline development, unless there was an explicit mention of involvement in systematic reviews. Systematic reviews that focussed on synthesising the evidence related to stakeholder involvement in primary research were also excluded. We excluded titles without abstracts and review protocols; this was a pragmatic decision made in light of the high volume of search results.
Definition of key terms
We used the following operational definitions, pre-stated in the protocol , to support the application of the selection criteria:
Stakeholder—any person who would be a knowledge user of research but whose primary role is not directly in research. Potential stakeholders include a broad range of people, including those who are actual or potential recipients of health or social care, where this may include patients, carers and family members, or people interested in remaining healthy who are seeking information about a health condition or treatment for personal use ; members of organisations that represent people who use services; people with a professional role in health and social care; policy makers and managers. We documented the types of people involved within any evidence included in this review, highlighting where this included patients, carers and family members, and where this included other stakeholders only.
Systematic review—a research process in which literature relevant to a stated question is identified and brought together (synthesised) using explicit methods , including reporting of inclusion/exclusion criteria, search methods and details of included studies. We accepted systematic reviews regardless of the type of evidence synthesised (i.e. quantitative, qualitative, mixed-methods) and the type of question addressed (e.g. intervention effectiveness, diagnostic test accuracy, patient experiences).
Involvement in a systematic review—any role or contribution of stakeholders toward the development of a review protocol, completion of any of the stages of a systematic review or dissemination of the findings of a review.
Methods of applying selection criteria
One review author (PC) ran the search strategy and excluded any obviously irrelevant titles. Two reviewers (PC, AP) independently reviewed the abstracts and applied selection criteria to the first 500 records; agreement was explored and a full team discussion held to clarify the selection criteria. This clarification led to a number of post hoc exclusion criteria (described above under selection criteria and within Additional file 1). Subsequently, we agreed that two independent review authors (PC, AP) should review a further 500 records using the clarified criteria and that if agreement between independent reviewers was greater than 95% when using these refined criteria, then subsequent selection of papers would be performed by one reviewer only; this agreement was 97%, and therefore, one reviewer (AP) screened the remaining abstracts. The full papers from abstracts included after the screening process were considered at the data extraction and judgement stages (see below); if a paper was found not to meet the inclusion criteria at this stage, it was excluded.
Data extraction and synthesis
For all included papers, one reviewer (AP) extracted and categorised data into structured tables. Extracted data included bibliographic information, type of paper, stated aim, topic/focus of systematic review, study/review methodology, description of reported involvement, details of people involved, stage in review process at which people involved and any formal research methods used. Retrospective categorisation of data included focus of review and type of evidence synthesised (see Additional file 1, protocol deviations). Details of the operationalisation of these data extraction items are provided in Additional file 4.
Judgement of comprehensiveness of description
Our review aim was to describe key features of how stakeholders have been involved in systematic reviews; consequently, we were principally concerned with the comprehensiveness of the description of methods of involvement, rather than appraising the quality of the methods of the reviews. We devised a method for judging the comprehensiveness of the description of the method or approach to involvement, given that there are no standardised tools for such a task. Criteria for categorising the comprehensiveness of the description provided within papers were developed, adapted from Pollock . Initially, two reviewers (AP, CS) assigned these criteria independently for a random sample of 20% of papers identified from step 1 of searching; this was 42 of 210 papers. There was agreement between independent reviewers for 57% (24/42) of the assessed sample. The agreement between reviewers, implications relating to disagreements and perceived risk of bias to the review results are reported in Additional file 5. Following discussion and clarification of criteria (see Additional file 5), it was agreed that one reviewer would assign judgements to the remaining papers, using the following criteria:
‘Green’—comprehensive description of one (or more) specific method or approach to the involvement of stakeholders in systematic reviews. Description sufficient to enable replication of methods
‘Amber’—a brief or partial description of one (or more) specific method or approach to the involvement of stakeholders in systematic reviews. Description sufficient to enable partial replication of methods
‘Red’—few details provided and/or inadequate description of the method or approach of involvement of stakeholders in systematic reviews. Description insufficient to enable any replication of methods
Detailed description of methods or approaches to involvement
Additional, more detailed, data extraction was performed for papers that were judged as ‘green’ for comprehensiveness of description. In addition to a narrative description of the methods or approaches to involvement, one reviewer (AP) extracted and tabulated the stated aim of involvement, number and characteristics of people involved, methods of recruitment, format of involvement (e.g. face-to-face meeting, telephone meeting, written consultation, online survey), amount of involvement (number of meetings, number of days involved), details of ethical approval and financial compensation given to stakeholders, evaluation of the involvement and tools used for reporting involvement.
Stakeholder involvement in this systematic review
One consumer (HG) and two consumer representatives (RM, CS) were members of the project and author team for this systematic review. All contributed to face-to-face discussions which led to the development of the review protocol, and read, commented on and had authorship of the published protocol. All contributed to project teleconferences throughout the review, particularly when making decisions relating to the stepwise search methods. Additionally, CS independently applied judgements of comprehensiveness to a sample of full papers. All three discussed the key findings of this review and contributed as authors to the final manuscript.
Results of the search
We screened 12,908 titles and abstracts and applied selection criteria to 672 full papers. Three hundred sixty-nine of these 672 full papers were excluded: 118 as they were abstracts only, 18 as they were protocols, 16 as they were duplicates and 217 as they did not meet our inclusion criteria. Reasons that these 217 did not meet our inclusion criteria are listed in table of excluded studies (Additional file 6); main reasons for exclusion were that the paper was a systematic review but there was no involvement of people (approximately 30%), the paper did not describe or report a systematic review (approximately 25%) or the paper described involvement in research other than a systematic review (approximately 25%). This left 291 papers that met our criteria for inclusion in the scoping review (see Fig. 1).
Characteristics of included papers
Details of the 291 included papers are provided in the table of included studies (Additional file 7). A brief summary is described below.
Type of paper
Thirty-one percent of included papers were published systematic reviews; 54% were reports of a guideline or recommendation in which a systematic review component was described; and 5% were papers specifically describing methods of involving stakeholders in a systematic review.
Thirty percent of the included papers involved patients and/or carers within the systematic review process, while 41% involved other stakeholders (e.g. health professionals, academic experts, representatives of patient organisations) but not patients or their family members. In almost one third of the included papers (29%), it was not clear who the stakeholders involved in the review were and whether this included patients and/or carers.
One third (31.6%) of papers were from the USA, one quarter (26.1%) from the UK and 10.0% from Canada. Of the remaining papers, 22.7% were from Australia, Netherlands, Germany, Italy, France or Spain, and 9.6% from a further 15 countries with between 1 and 4 papers each (see Table 1).
Stage of the review process
In almost half of the papers (47.8%), the stage of the review process at which stakeholders were involved was unclear. In just over one quarter (27.5%), stakeholders were involved in interpreting the results after the evidence had been synthesised. In around one fifth (22.3%), stakeholders were involved either throughout the whole review process or during one or more stages of review completion (see Table 2).
Focus of the review
Seventy-one percent of the included systematic reviews were judged to be focussed on one of the International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10) categories (Table 3 and table of included studies (Additional file 7)). Most frequently (10%), this was ‘factors influencing health status and contact with health services’, where reviews covered topics such as the effectiveness or implementation of care pathways for specific population (e.g. paediatrics, geriatrics, emergency care). The specific diseases or health areas covered by the greatest numbers of reviews were mental and behavioural disorders (8.6%), neoplasms (6.9%), diseases of the musculoskeletal system and connective tissue (6.2%) and certain infectious and parasitic diseases (5.5%). Thirteen percent of the reviews which did not fit one of the ICD-10 categories were focussed on a specific intervention, most commonly medical or surgical interventions (8.6%) and public health interventions (5.2%). Ten percent of reviews were focussed on an area of research, rather than a specific health or disease area or intervention; more than half of these (55%, n = 29) were focussed on methods of stakeholder involvement or engagement, while the remainder focussed on other areas of research methods, such as methods of statistical tests within primary research. The remaining 7% were unable to be categorised within any of these groups and focussed on, for example, areas such as teaching, data protection and criminal justice.
Comprehensiveness of description of method or approach to involvement
Table 4 shows the assigned judgements of the comprehensiveness of the description of the method or approach to involvement. Figure 2 illustrates the proportion of different types of paper which were judged to be ‘green’, ‘amber’ or ‘red’, when patients/carers were involved and at different stages in the review process.
In total, 59% of the included papers were judged to provide few or inadequate details (‘red’), with only 10% judged to provide a comprehensive description of one, or more, method or approach to involvement (‘green’).
Detailed description of methods or approaches to involvement
The 30 papers which were judged as providing a comprehensive (‘green’) description of their methods or approaches to involvement included 14 ‘methods’ papers describing an experience of stakeholder involvement in one (or more) systematic review [25, 29,30,31,32,33,34,35,36,37,38,39,40,41]; 11 systematic reviews in which the stakeholder involvement was concurrently described [42,43,44,45,46,47,48,49,50,51,52]; 2 guidelines or clinical recommendations, in which the involvement in the systematic review component was described [53, 54]; and 1 paper which described the development of a tool to report stakeholder involvement . Two of the papers each described two different systematic reviews [37, 55], meaning that there are a total of 32 systematic review described. Table 5 summarises the key characteristics of these 32 systematic reviews, and Table 6 summarises the data relating to stakeholder involvement and a brief narrative summary of key features is provided below.
Table 5 states the aim and focus of the 32 systematic review. Sixty-eight percent were focussed on one of the ICD-10 categories; with mental and behavioural disorders being the most common health topic (22%). Sixteen percent were focussed on a specific intervention rather than a disease area, most commonly on a public health intervention (12%). The remaining 16% were focussed on either research or another topic.
A majority of the reviews (56%) synthesised both qualitative and quantitative evidence, while 19% only included quantitative studies and 12.5% only included qualitative studies. The type of evidence included was unclear for 12.5%. Two of the reviews described using a ‘realist’ review methodology, and 2 were Cochrane reviews of randomised controlled trials.
Seventy-eight percent of the systematic reviews involved patients, carers or family members, while in one (3%), the people involved were peer support workers. In 19% of systematic reviews, the only people involved were professionals or academic experts, although one of these  aimed to recruit patient representatives, but failed to do so. Where there were face-to-face meetings, the number of stakeholders involved ranged from 2 to 27; where there were one-off events, often advertised as open to the general public, the numbers of stakeholders involved ranged from 15 to 81; where involvement did not require a face-to-face meeting, for example using an electronic Delphi or survey, the numbers invited ranged from 29 to 340 (see Table 6).
Geographical location (from which stakeholders were recruited)
The majority of the involvement occurred in the UK, with two thirds (66%) of papers describing UK-based activities. Of the remaining 34%, 2 recruited people from across Europe, 3 were carried out in Canada, 3 in the Netherlands and 1 in Australia, USA and Spain.
How people were recruited
For 69% (22/32) of the systematic reviews, people were personally invited to be involved. This involved personal invitations to known people (in 12/22; [29, 31, 39, 41,42,43,44, 48, 54, 55, 57]); personal invitations to an existing group or groups (in 6/22; [34, 35, 37, 46, 58]); or purposive sampling, using similar methods as sampling for qualitative research (in 4/22; [32, 38, 40, 56]). For a further 7/32 of the systematic review, involvement opportunities were advertised to the general population, often snowballing information out via target groups and organisations, and anyone who volunteered could get involved [25, 36, 47, 49, 50, 52, 53]. A combination of different recruitment strategies was used for 1 systematic review , and the method of recruitment was unclear for 3 systematic reviews [30, 45, 51].
Format of involvement
The format of involvement comprised direct, face-to-face interaction in 81% and an electronic Delphi method or survey in 19% of the systematic reviews. The face-to-face interaction was either in the format of a meeting (53%; [29, 31, 32, 34, 35, 38, 40,41,42,43,44, 46, 50, 52, 55, 58]), a larger workshop or public event (19%; [25, 36, 37, 47, 49]) or a combination of both of these (9%; [33, 45, 48]). In each of the 6/32 systematic reviews which used an electronic Delphi method, there was a specific and focussed aim of stakeholder involvement; in 4/32 [30, 53, 54, 56], this was broadly related to reaching consensus on factors, recommendations or statements arising from the results of the systematic review, and in 2/32 [39, 55], this was to reach consensus on the topic or focus of the systematic review.
Amount of involvement
Where there was direct face-to-face interaction, there could be between 1 and 20 meetings or events. The majority (83%) of the 24 reviews providing this information held 4 or less meetings (median 2 meetings), while one held 5 meetings plus 3 public workshops . Three held multiple meetings (12, 15 and 20 respectively by [25, 46, 58]); in each of these three examples, the approach is described as ‘participatory’. Where reported, the length of face-to-face meetings varied from 1 h to ½ day. Generally, the Delphi approach involved three rounds of an electronic survey, although in one example after two rounds of Delphi voting there was a direct face-to-face consensus meeting .
Details of ethical approval were reported for 31% of systematic reviews; for details, see Table 6. One paper reported that ethical approval was sought but not required . No details relating to ethical approval were provided by the remaining 66% of papers.
Expenses (such as travel, accommodation and care costs for family members) were reported to be paid to people involved in 25% of systematic reviews; in two, this was expenses only, while in six money or a voucher was provided in addition to expenses (see Table 6). No details relating to financial compensation are reported in the remaining 75% of systematic reviews.
Tools or method of reporting involvement
Thirty-four percent of the included papers had a clear method of reporting involvement. Four used some sort of tool, framework or checklist: Concannon et al.  developed and used a 7-item question for reporting stakeholder involvement in research, Liabo  used a framework for considering impact of involvement, Martin et al.  reported an evaluation based on reporting standards within Guidance for Reporting Involvement of Patients and the Public (GRIPP) checklist  and Saan et al.  developed and used a Tool for Recording and Accounting for Stakeholder Involvement (TRASI). Seven specifically reported activities which groups of stakeholders were involved in within a specific section of text, tables or supplementary files [30, 35, 37, 38, 47, 50]. Within the remaining 66% of systematic reviews, information about methods of involvement was not reported within a particular section, table or file, but was distributed throughout the paper.
Evaluation of the methods of involvement
None of the 32 studies carried out any formal evaluation of the impact of involving stakeholders; however, 28% collected data relating to the views and experiences of people involved. Of these, four used a questionnaire to elicit the views and experiences of stakeholders [29, 30, 36, 41]; three held a discussion with stakeholders in which they were encouraged to share or reflect on their experiences and perspectives [31, 33, 47]; and two had both a questionnaire and a discussion [38, 44]. In addition, Liabo  reported data arising from audio recordings and minutes of all meetings, Hyde et al.  described ‘impact’ within a table, while the reflections of the researchers on the process of involvement were discussed by others [35, 37, 40, 52].
Key findings: evidence-base relating to stakeholder involvement in systematic reviews
We identified 291 papers describing stakeholder involvement in systematic reviews. Approximately two thirds of published examples describe UK activities, but we found examples from at least 24 countries. Reporting of who was involved, in what ways and at what stage in the review process was generally very poor, and the majority of the papers (59%) were judged to provide few details and/or an inadequate description of the method or approach of involving stakeholders. Thirty percent of systematic review teams clearly involved patients/carers, but in many cases (41%), the stakeholders involved health professionals, academic experts or representatives of patient organisations, but not patients or their family members.
We identified 30 papers, describing 32 systematic reviews, which we judged to have sufficiently comprehensive reporting to allow a more in-depth synthesis of methods or approaches to the involvement of stakeholders in systematic review. We have described key features of how stakeholders have been involved in systematic reviews, using data from these 32 examples. However, it was notable that, despite the selection of systematic reviews which were judged to provide a comprehensive description of one or more method of involvement, there was still inadequate (or absence of) reporting of a number of features in which we were interested. For example, the majority of papers did not provide any information relating to ethical approval or financial compensation to the stakeholders involved. A key contributing factor to the poor reporting relating to aspects of how stakeholders were involved may have been the lack of a tool or standardised method for reporting. On the few occasions where a particular tool has been used to support reporting of information relating to involvement, the tool has often been developed specifically by the systematic review authors. In many cases, the method of reporting comprises a written description of the activities in which stakeholders have been involved, but we found inconsistencies in the type of information presented and the location of this information within published papers.
Implications: methods of involving stakeholders in systematic reviews
The evidence which we have synthesised demonstrates that actively involving stakeholders within systematic reviews is feasible, and can be incorporated into a wide range of different types of systematic review. While there can be considerable variation in how stakeholders are involved, and the types of stakeholders who are involved, and there is currently an absence of evidence to directly inform choices for methods of stakeholder involvement within future reviews, a number of implications can be drawn from our synthesised evidence. In particular, evidence drawn from the 32 examples explored in this review can highlight some of the methodological decisions which may be made when planning stakeholder involvement in future reviews. These include:
Will people directly affected by the healthcare topic addressed within the systematic review (i.e. individual patients, carers or family members) be involved? Will health professionals, academic experts or representatives from patient organisations be involved?
How to find people to involve? Within our 32 examples, we found two key methods of recruiting stakeholders to be in systematic reviews; in the majority of our examples, there were personal invitations to known individuals or groups, but in some cases, recruitment occurred through advertising to the general population in order to get stakeholders to volunteer to be involved.
How will people be involved? Within our 32 examples, two distinct methods of involving people in a systematic review were identified: (i) face-to-face meetings or events or (ii) electronic Delphi method. Where there were face-to-face meetings, these could be attended by invited participants only or could be an open event or workshop to which members of the public are invited to attend. Invited participants may only attend a small number (often between 1 and 4) of meetings during the course of a systematic review, but this may be much more where a participatory approach is used.
How many stakeholders to involve? The current evidence base indicates that the number of stakeholders depends on the way in which they will be involved. Evidence from the 291 papers in our synthesis shows that 1 stakeholder may be a co-author on a systematic review, 2–10 stakeholders may be members of a steering group, 5–50 stakeholders may attend face-to-face meetings or focus groups and 20–400 stakeholders may participate in Delphi rounds or attend events or conferences.
Use of research methods? Our examples highlighted that the following research methods have sometimes been incorporated into stakeholder involvement in systematic reviews: focus groups, interviews and a number of consensus decision-making techniques such as Delphi, Nominal Group Technique and voting/ranking processes.
Other issues to consider when planning stakeholder involvement in systematic reviews are whether ethical approval will be required and resources for payment of expenses and any other financial compensation or reward.
Although there is insufficient evidence to directly inform choices relating to who to involve and in what way, the findings arising from the 32 papers identified in this review have been used to produce, in collaboration with Cochrane Training, freely available online learning material and resources . There have been many urgent calls for high-quality training materials, reporting guidelines and examples of best practice to support active stakeholder involvement and to enhance the relevance, usefulness and accessibility of systematic reviews [2, 16, 18, 33, 61]; the evidence from this review therefore can arguably currently play a key role in learning and support relating to active stakeholder involvement in systematic reviews.
Implications: reporting stakeholder involvement in systematic reviews
Recording and reporting of stakeholder involvement is important, both to ensure transparency in relation to the contributions and roles of different stakeholders within the review process and to contribute to the evidence base relating to this field. This scoping review highlights that the current reporting of involvement in systematic reviews is very poor and sometimes absent, and rarely provides a comprehensive description of who was involved and in what way. While there are a number of tools and frameworks which review authors could consider using (e.g. [36, 46, 55]), there is not currently any tool, guidance or recommendations specifically designed to support reporting of involvement within systematic reviews. Generic guidance relating to the reporting of stakeholder involvement in research has recently been updated (GRIPP2, ); however, this guidance has not been specifically tested for use with systematic reviews and has lacked international input during development. It is clear that there is an urgent need for improved reporting of involvement of stakeholders in systematic reviews. Such reporting should enhance the ability to develop evidence-based guidance around how to involve stakeholders in systematic reviews, and to explore and evaluate the impact of involvement.
Identification of relevant systematic reviews and data extraction
It is unlikely we identified all relevant examples of stakeholder involvement in systematic reviews, as we adopted a pragmatic search approach aimed at efficiency within project time and resource constraints. This was compounded by poor reporting and inconsistent terminology in this area. We believe it is highly likely that there are many systematic reviews where stakeholders played a key role that our methods could not identify. Our decision to exclude titles without abstracts and review protocols at the study selection stage may have introduced publication bias into our results, with a bias toward inclusion of papers published in peer-reviewed journals. Only one review author extracted data from the included studies, and there is the potential that this may have introduced bias and errors in extraction. In an attempt to improve transparency and reduce data extraction errors, we copied and pasted data verbatim from included papers into an electronic data extraction sheet. This is reported in the table of included studies (Additional file 7).
Judgement to identify those with most comprehensive description
The agreement between independent reviewers when applying the ‘comprehensiveness’ judgement to a subset of papers indicated that there were disagreements on around 17% of ‘green’ categorisations. We did not have the time or resources to have independent judgement on a higher proportion of studies. We are therefore not confident that our subset includes all papers which may provide an adequate description of some parts of the methods of involving people in a review. However, as the aim of this phase was to identify and describe methods of involvement from examples of systematic reviews, the impact of potentially falsely including or excluding a paper from this subset was perceived to be low. We present the included ‘green’ papers as examples of systematic reviews in which there was involvement of stakeholders and take care to stress that these are examples rather than a comprehensive sample.
Our judgement of the comprehensiveness of the description of the methods was not a judgement of the quality of the involvement methods and only relates to the depth of the description of stakeholder involvement provided in the identified paper. Over half (54%) of the 291 included papers were reports of a guideline or recommendation, but only 2 of these were judged as ‘green’ for comprehensiveness of description. A potential explanation for this finding could be that stakeholder involvement is generally a core component of guideline development, but the primary focus of related journal publications is often the key clinical messages and implications, rather than the methods of the guideline, which are often fully described elsewhere. A judgement of ‘amber’ or ‘red’ for the comprehensiveness of the description of the method of involvement in the published paper is not an indication either that the quality of the methods was poor or that details of methods of stakeholder involvement are not available elsewhere.
This systematic review summarises evidence relating to the involvement of stakeholders in systematic reviews. We identified a relatively large number (291) of papers reporting stakeholder involvement in systematic reviews, but the quality of reporting was generally very poor. The level of reporting of involvement of stakeholders in systematic reviews, and the inconsistencies in which this is reported, must be improved so that guidance around how people can be involved in systematic reviews can be developed and the impact of involvement explored. This scoping review lends support to calls for high-quality training materials and examples of best practice to support active patient and public involvement and enhance the relevance, usefulness and accessibility of systematic reviews [2, 16, 18, 61, 63]. We identified a subset of 30 papers which we judged to provide a comprehensive description of stakeholder involvement in systematic reviews, and used these examples to summarise different ways in which stakeholders have been involved in systematic reviews. These examples arguably currently provide the best available information to inform and guide decisions around the planning of stakeholder involvement within future systematic reviews. This evidence has been used by Cochrane Training to develop online learning resources relating to how to involve people in systematic reviews , and has been used to develop a framework for describing stakeholder involvement in systematic reviews (Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, Morris J, Watts C, Morley R: Development and application of a framework to describe how stakeholders have been involved in systematic reviews, submitted).
Authors and Consumers Together Impacting on eVidencE
Allied and Complementary Medicine Database
Cochrane Database of Systematic Reviews
Cochrane Central Register of Controlled Trials
Cumulative Index to Nursing and Allied Health Literature
Database of Abstracts of Reviews of Effectiveness
Database of Promoting Health Effectiveness Reviews
An online platform for accessing research databases
Excerpta Medica Database
Evidence for Policy and Practice Information and Co-ordinating Centre
Guidance for Reporting Involvement of Patients and the Public
Health Technology Assessment
International Statistical Classification of Diseases and Related Health Problems 10th Revision
On-line Medical Library database
An online platform for accessing research databases
(‘Pretty darn quick’) database of systematic reviews of health systems evidence
Camden C, Shikako-Thomas K, Nguyen T, Graham E, Thomas A, Sprung J, Morris C, Russell DJ. Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts. Disabil Rehabil. 2015;37:1390–400.
INVOLVE. Exploring the impact of public involvement on the quality of research: examples. Eastleigh: INVOLVE; 2013.
Kreis J, Puhan MA, Schunemann HJ, Dickersin K. Consumer involvement in systematic reviews of comparative effectiveness research. Health Expect. 2013;16:323–37.
Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise J-M, Santa J, Conway PH, Daudelin D, Morrato EH, Leslie LK. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. 2012;27:985–91.
Heaton J, Day J, Britten N. Collaborative research and the co-production of knowledge for practice: an illustrative case study. Implement Sci. 2016;11:20.
Boote J, Wong R, Booth A. ‘Talking the talk or walking the walk?’ A bibliometric review of the literature on public involvement in health research published between 1995 and 2009. Health Expect. 2015;18:44–57.
CIHR. Canadian Institutes of Health Research (CIHR): strategy for patient-oriented research (SPOR) - patient engagement. Ottawa: CIHR; 2014.
Hickey G, Richards T, Sheehy J. Co-production from proposal to paper. Nature. 2018;562:29-31.
NHMRC. National Health and Medical Research Council (NHMRC): a model framework for consumer and community participation in health and medical research. Canberra: NHMRC; 2005.
Nasser M, Clarke M, Chalmers I, Brurberg KG, Nykvist H, Lund H, Glasziou P. What are funders doing to minimise waste in research? Lancet. 2017;389:1006–7.
Wallace J, Nwosu B, Clarke M. Barriers to the uptake of evidence from systematic reviews and meta-analyses: a systematic review of decision makers’ perceptions. BMJ Open. 2012;2:e001220.
Cochrane Consumer Network. http://consumers.cochrane.org/. Accessed 6 Nov 2018.
INVOLVE. Briefing notes for researchers: involving the public in NHS, public health and social care research. Eastleigh: INVOLVE; 2012.
Horey D. Consumer involvement in the Cochrane Collaboration: background paper (strategic session). Auckland: Cochrane Consumer Network; 2010.
Cochrane Consumer Network. Cochrane consumer network structure and function review, 2015. https://consumers.cochrane.org/ccnet-structure-and-function-review. Accessed 6 Nov 2018.
Boote J, Baird W, Sutton A. Public involvement in the systematic review process in health and social care: a narrative review of case examples. Health Policy. 2011;102:105–16.
Morley RF, Norman G, Golder S, Griffith P. A systematic scoping review of the evidence for consumer involvement in organisations undertaking systematic reviews: focus on Cochrane. Res Involv Engagem. 2016;2:36.
Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, Brito JP, Boehmer K, Hasan R, Firwana B, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14:89.
ACTIVE: Authors and Consumers Together Impacting eVidencE. http://training.cochrane.org/ACTIVE. Accessed 6 Nov 2018.
Pollock A, Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, Watts C, Morley R. Stakeholder involvement in systematic reviews: a protocol for a systematic review of methods, outcomes and effects. Res Involv Engagem. 2017;3:9.
Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.
Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.
Colquhoun HL, Levac D, O'Brien KK, Straus S, Tricco AC, Perrier L. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67:1291-4.
Jepson R, Harris F, MacGillivray S, Kearney N, Roaw-Dewar N. A review of the effectiveness of interventions, approaches and models at individual, community and population level that are aimed at changing health outcomes through changing knowledge attitudes and behaviour; 2007.
Harris J, Croot L, Thompson J, Springett J. How stakeholder participation can contribute to systematic reviews of complex interventions. J Epidemiol Community Health. 2016;70:207–14.
Cochrane Consumer Network. http://consumers.cochrane.org/healthcare-users-cochrane. Accessed 6 Nov 2018.
Evidence-informed policy and practice (EPPI) centre: what is a systematic review? http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=67. Accessed 6 Nov 2018.
Pollock A, Campbell P, Deery R, Fleming M, Rankin J, Sloan G, Cheyne H. A systematic review of evidence relating to clinical supervision for nurses, midwives and allied health professionals. J Adv Nurs. 2017;10:1825–37.
Bayliss K, Starling B, Raza K, Johansson EC, Zabalan C, Moore S, Skingle D, Jasinski T, Thomas S, Stack R. Patient involvement in a qualitative meta-synthesis: lessons learnt. Res Involv Engagem. 2016;2. https://doi.org/10.1186/s40900-016-0032-0.
Boelens PG, Taylor C, Henning G, Marang-van de Mheen PJ, Espin E, Wiggers T, Gore-Booth J, Moss B, Valentini V, van de Velde CJ. Involving patients in a multidisciplinary European consensus process and in the development of a ‘patient summary of the consensus document for colon and rectal cancer care’. Patient. 2014;7:261–70.
Braye S, Preston-Shoot M. Emerging from out of the shadows? Service user and carer involvement in systematic reviews. Evid Policy. 2005;1:173–94.
Bunn F, Sworn K, Brayne C, Iliffe S, Robinson L, Goodman C. Contextualizing the findings of a systematic review on patient and carer experiences of dementia diagnosis and treatment: a qualitative study. Health Expect. 2015;18:740–53.
Coon JT, Gwernan-Jones R, Moore D, Richardson M, Shotton C, Pritchard W, Morris C, Stein K, Ford T. End-user involvement in a systematic review of quantitative and qualitative research of non-pharmacological interventions for attention deficit hyperactivity disorder delivered in school settings: reflections on the impacts and challenges. Health Expect. 2016;19:1084–97.
Hyde C, Dunn KM, Higginbottom A, Chew-Graham CA. Process and impact of patient involvement in a systematic review of shared decision making in primary care consultations. Health Expect. 2017;20:298–308.
Jamal F, Langford R, Daniels P, Thomas J, Harden A, Bonell C. Consulting with young people to inform systematic reviews: an example from a review on the effects of schools on health. Health Expect. 2015;18:3225–35.
Martin S, Fleming J, Cullum S, Dening T, Rait G, Fox C, Katona C, Brayne C, Lafortune L. Exploring attitudes and preferences for dementia screening in Britain: contributions from carers and the general public. BMC Geriatr. 2015;15:110.
Oliver K, Rees R, Brady LM, Kavanagh J, Oliver S, Thomas J. Broadening public participation in systematic reviews: a case example involving young people in two configurative reviews. Res Synth Methods. 2015;6:206–17.
Pollock A, Campbell P, Baer G, Choo PL, Morris J, Forster A. User involvement in a Cochrane systematic review: using structured methods to enhance the clinical relevance, usefulness and usability of a systematic review update. Syst Rev. 2015;4:55.
Serrano-Aguilar P, Trujillo-Martin MM, Ramos-Goni JM, Mahtani-Chugani V, Perestelo-Perez L, Posada-de la Paz M. Patient involvement in health research: a contribution to a systematic review on the effectiveness of treatments for degenerative ataxias. Soc Sci Med. 2009;69:920–5.
Stewart R, Oliver S. What is known about communication with parents about newborn bloodspot screening? In: Carr S, Coren E, editors. Collection of examples of service user and carer participation in systematic reviews. London: Social Care Institute for Excellence; 2003.
Vale C, Nightingale A, Spera N, Whelan A, Hanley B, Tierney JF. Late complications from chemoradiotherapy for cervical cancer: reflections from cervical cancer survivors 10 years after the national cancer institute alert. Clin Oncol (R Coll Radiol). 2010;22:588–9.
Concannon TW, Fuster M, Saunders T, Patel K, Wong JB, Leslie LK, Lau J. A systematic review of stakeholder engagement in comparative effectiveness and patient-centered outcomes research. J Gen Intern Med. 2014;29:1692–701.
Edwards D, Evans N, Gillen E, Longo M, Pryjmachuk S, Trainor G, Hannigan B. What do we know about the risks for young people moving into, through and out of inpatient mental health care? Findings from an evidence synthesis. Child Adolesc Psychiatry Ment Health. 2015;9:55.
Hayden JA, Killian L, Zygmunt A, Babineau J, Martin-Misener R, Jensen JL, Carter AJ. Methods of a multi-faceted rapid knowledge synthesis project to inform the implementation of a new health service model: collaborative emergency centres. Syst Rev. 2015;4:7.
Higginson IJ, Evans CJ, Grande G, Preston N, Morgan M, McCrone P, Lewis P, Fayers P, Harding R, Hotopf M, et al. Evaluating complex interventions in end of life care: the MORECare statement on good practice generated by a synthesis of transparent expert consultations and systematic reviews. BMC Med. 2013;11:111.
Liabo K. Service user involvement in research: collaborating on a systematic review with young people who have experience of being in care. London: University of London, Institute of Education; 2013.
Liu J, Davidson E, Bhopal R, White M, Johnson M, Netto G, Deverill M, Sheikh A. Adapting health promotion interventions to meet the needs of ethnic minority groups: mixed-methods evidence synthesis. Health Technol Assess. 2012;16(44). https://doi.org/10.3310/hta16440.
McConachie H, Parr JR, Glod M, Hanratty J, Livingstone N, Oono IP, Robalino S, Baird G, Beresford B, Charman T, et al. Systematic review of tools to measure outcomes for young children with autism spectrum disorder. Health Technol Assess. 2015;19:1–506.
McCusker J, Yaffe M, Sussman T, Kates N, Mulvale G, Jayabarathan A, Law S, Haggerty J. Developing an evaluation framework for consumer-centred collaborative care of depression using input from stakeholders. Can J Psychiatr. 2013;58:160–8.
Pearson M, Chilton R, Wyatt K, Abraham C, Ford T, Woods HB, Anderson R. Implementing health promotion programmes in schools: a realist systematic review of research and experience in the United Kingdom. Implement Sci. 2015;10:149.
Rees R, Kavanagh J, Burchett H, Shepherd J, Brunton G, Harden A, Thomas J, Oliver S, Oakley A. HIV health promotion and men who have sex with men (MSM): a systematic review of research relevant to the development and implementation of effective and appropriate interventions. London: EPPI-Centre; 2004.
Smith E, Ross F, Donovan S, Manthorpe J, Brearley S, Sitzia J, Beresford P. Service user involvement in nursing, midwifery and health visiting research: a review of evidence and practice. Int J Nurs Stud. 2008;45:298–315.
Bond KS, Chalmers KJ, Jorm AF, Kitchener BA, Reavley NJ. Assisting Australians with mental health problems and financial difficulties: a Delphi study to develop guidelines for financial counsellors, financial institution staff, mental health professionals and carers. BMC Health Serv Res. 2015;15:218.
Oosterkamp BC, van der Sanden WJ, Frencken JE, Kuijpers-Jagtman AM. Caries preventive measures in orthodontic practice: the development of a clinical practice guideline. Orthod Craniofacial Res. 2016;19:36–45.
Saan MC, Boeije HR, Sattoe JN, Bal MI, Missler M, van Wesel F. Recording and accounting for stakeholder involvement in systematic reviews. Health Inf Libr J. 2015;32:95–106.
McGinn CA, Gagnon M-P, Shaw N, Sicotte C, Mathieu L, Leduc Y, Grenier S, Duplantie J, Abdeljelil AB, Légaré F. Users’ perspectives of key factors to implementing electronic health records in Canada: a Delphi study. BMC Med Inform Decis Mak. 2012;12:105.
Hesselink G, Kuis E, Pijnenburg M, Wollersheim H. Measuring a caring culture in hospitals: a systematic review of instruments. BMJ Open. 2013;3:e003416.
Morgan H, Hoddinott P, Thomson G, Crossland N, Farrar S, Yi D, Hislop J, Moran VH, MacLennan G, Dombrowski SU, et al. Benefits of incentives for Breastfeeding and Smoking cessation in pregnancy (BIBS): a mixed-methods study to inform trial design. Health Technol Assess. 2015;19:1–522 vii–viii.
Staniszewska S, Brett J, Mockford C, Barber R. The GRIPP checklist: strengthening the quality of patient and public involvement reporting in research. Int J Technol Assess Health Care. 2011;27:391–9.
Involving people: a learning resource for systematic review authors. http://training.cochrane.org/involving-people. Accessed 6 Nov 2018.
Dudley L, Gamble C, Allam A, Bell P, Buck D, Goodare H, Hanley B, Preston J, Walker A, Williamson P, Young B. A little more conversation please? Qualitative study of researchers’ and patients’ interview accounts of training for patient and public involvement in clinical trials. Trials. 2015;16:190.
Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, Altman DG, Moher D, Barber R, Denegri S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453.
Hoddinott P, Pollock A, O'Cathain A, Boyer I, Taylor J, MacDonald C, Oliver S, Donovan JL. How to incorporate patient and public perspectives into the design and conduct of research [version 1; referees: 2 approved, 2 approved with reservations]. F1000Research 2018;7:752. https://doi.org/10.12688/f1000research.15162.1.
Chilvers CED, Bagenal FS, Easton DF, Harris E, McElwain TJ. Survival of patients with breast cancer attending Bristol Cancer Help Centre. The Lancet, 1990;336:606–10
Goodare H (Editor). Fighting Spirit: the stories of women in the Bristol breast cancer survey. London; Scarlet Press, 1996.
The review has been funded by the Learning and Support Department and Consumer Network Co-ordinator, within Cochrane’s Central Executive Team. Findings from the review will be used to develop a learning resource to encourage and enable Cochrane Review Groups in involving consumers in Cochrane systematic reviews. Two authors are members of the funding body (RM and CW) and have been involved in conceptualising the design of the study and its outputs.
Availability of data and materials
All authors are involved with Cochrane (an international not-for-profit which publishes systematic reviews).
Alex Pollock is a senior research fellow at the Scottish Government funded ‘Nursing Midwifery and Allied Health Professions Research Unit’ (NMAHP RU) and has been involved in systematic reviews of complex interventions for over 15 years. She is an associate editor of the Cochrane Stroke Group and an author on 12 Cochrane reviews. She has implemented innovative ways of involving stakeholders within Cochrane reviews.
Pauline Campbell is a research fellow at NMAHP RU, with particular responsibility for completing systematic reviews relating to nursing and healthcare interventions, including rapid reviews commissioned by the Scottish Government’s Chief Nursing Officer. She is an author on 7 Cochrane reviews and has involved stakeholders in reviews using a range of methods.
Caroline Struthers is the education and training manager for the EQUATOR Network, responsible for developing educational resources, courses and training materials for a wide range of users, including researchers, editors, peer reviewers and the general public. Previous roles have included Information Specialist, Managing Editor and Training Coordinator for Cochrane, and she was also manager of an NHS-funded public engagement project for the Cochrane Dementia and Cognitive Impairment Group. At the time of this work, she was a member of the Cochrane Consumer Network Executive.
Anneliese Synnot is a research fellow and editor with Cochrane Consumers and Communication. Her research interests include innovations in evidence synthesis, both in terms of novel methods to involve consumers and others in the review production and dissemination and innovative review types, such as Living Systematic Reviews.
Jack Nunn is a researcher in the Public Health Department in the Centre for Health Communication and Participation at La Trobe University. He has recently worked on projects with Cochrane Australia, the World Health Organisation and the Australian Department of Health. He has previously worked for the UK National Institute of Health Research and Macmillan Cancer Support (UK) where he led the development and implementation of an internationally recognised model for building partnerships between the public and researchers.
Sophie Hill is the Head of the Centre for Health Communication and Participation (www.latrobe.edu.au/chcp), at La Trobe University, a centre she has established from the foundation of the Cochrane Consumers and Communication Review Group (http://cccrg.cochrane.org/). The Centre has an applied focus, with three roles: coordinating the production and publication of evidence on interventions to communicate with people about health; innovative research on communication issues that have been neglected, such as multimorbidity; and a knowledge translation function, for getting evidence into practice and policy.
Heather Goodare is a Cochrane consumer reviewer for breast cancer and stroke; she was the first patient representative on the BMJ Editorial Board (1995–1999) and is a Life Fellow of the Royal Society of Medicine. Originally an academic book editor, Heather trained as a counsellor after her own experience of breast cancer and of a flawed research study (Chilvers et al. ) in which she was a patient (see Goodare ).
Chris Watts is the Learning and Support Officer for Cochrane Training. He works on a range of Cochrane learning projects including design and development of learning materials and pathways to support learners in a variety of Cochrane roles, particularly through online resources and initiatives for distance learners. Chris is a researcher by background and previously worked at the Royal College of Nursing in the UK, where he led a team of Research Analysts delivering evidence synthesis and evaluation projects supporting professional development and policy.
Richard Morley is the Consumer Coordinator for Cochrane, supporting consumer involvement in the production and dissemination of Cochrane evidence. He has extensive experience of public engagement and partnership working in the voluntary, public and education sectors.
Jacqui Morris is a Reader in Rehabilitation Research with a particular interest in how systematic review evidence can be implemented in allied health professionals practice. She is a co-author on several Cochrane reviews.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Protocol deviations. (DOCX 31 kb)
Search strategy. (DOCX 23 kb)
Year of publication of included studies. (DOCX 25 kb)
Data extraction (scoping review). (DOCX 35 kb)
Agreement between independent reviewers for judgements. (DOCX 32 kb)
Table of excluded studies. (DOCX 50 kb)
Table of included studies. (XLSX 166 kb)