Table 6 Summary of data relating to stakeholder involvement in ‘green’ systematic reviews
From: Stakeholder involvement in systematic reviews: a scoping review
Review | Description of involvement | People involved | Method of involvement | Formal research methods | Ethical approval? | Financial compensation (or alternative) for people involved? |
|---|---|---|---|---|---|---|
Bayliss et al. [29] | ‘Patient research partners’ contributed via teleconference calls, written (email) and one face-to-face meeting. Teleconferences: during completion of searches and paper selection, to input into search By email: 3 volunteers helped with coding framework to coproduce themes with researchers Meeting: invited to attend a 90-min face-to-face focus session at an annual meeting | Women with rheumatoid arthritis (n = 8); all with understanding of the research process | ‘Patient research partners’—teleconferences, emails and invitation to one 90-min meeting | – | No information provided | No information provided |
Boelens et al. [30] | Experts participated in three web-based online voting rounds, discussion and lectures at a conference. Patients and patient representatives were involved in the Delphi and had a role in developing a patient summary and future testing and implementation of this patient version. Drafts of the patient summary were circulated by email, and comments invited. | ‘Experts’ (professionals). People who were colon or rectal cancer survivors and representatives of this group. | Delphi approach (online, 3 rounds) ‘In the Delphi online voting process, 32 delegates were able to vote. Patient representatives had one vote. It is not known how many patients attended the meeting.’ | Delphi approach | No information provided | No information provided |
Bond et al. [53] | Statements arising from a systematic review were used to form a questionnaire that was administered to the expert panels via SurveyMonkey. The panel members were asked to rate each of the statements, using a 5-point scale, according to whether or not they thought the statement should be included in the guidelines. | Five expert panels: financial counsellors, financial institution staff, mental health professionals, mental health consumer advocates and carer advocates. All panellists had to be 18Â years or older, living in Australia, and have either professional or personal experience with mental health problems and financial difficulties. | Delphi approach (online, 3 rounds) (340 participants invited; 214 completed round 1, 170 completed round 2, 162 completed round 3) | Delphi approach | Ethical approval granted. | No information provided |
Braye and Preston-Shoot [31] | The first meeting aimed to seek views on the content and process of the study, finalising the research questions and concluding the protocol, and to explore and consider participants perspectives around the topic. The second meeting aimed to present and consider emerging findings and agree recommendations for the final report, and to consider the broader implications. | Service users and carers (n = 15), professionals (n = 16) | Two face-to-face meetings. Information was sent out prior to meetings, introductory presentation. | – | No information provided | In addition to researcher time to undertake these negotiations, money was also set aside to meet the costs of travel and special transport, accommodation and fees related to attendance........ . It was self-evident here that individuals, or their organisations, should be paid for their time, immediately in cash when required. |
Bunn et al. [32] | Focus groups and interviews with key stakeholders. Involvement occurred after completion of the systematic review, in order to confirm the key themes from the review. | 27 participants (three people with dementia, 12 carers, six dementia service providers and five older people without dementia). Purposive sampling approach to recruitment. | Four focus groups and three interviews | Focus groups, interviews | Ethical approval was obtained from NRES Committee East of England. REC reference 10/H0302/19. | Participants were given a £10 voucher in appreciation of their time, and their travel expenses were reimbursed. |
Concannon et al. [42] | Two meetings at different stages in the review process. Meeting 1 aimed at confirming the research questions and study design. Meeting 2 to review preliminary results. Group members also participated by email and phone, commenting on tables, figures and manuscript drafts, and were asked to assist in dissemination. In addition, 3 people (2 patients) contributed to review the planning stages. | Consumers, professionals, researchers (n = 7) | Two face-to-face meetings. Email and phone communication throughout the review. | – | No information provided | No information provided |
Coon et al. [63] | An Expert Advisory group—involved throughout the project, including commenting on the protocol, editing draft chapters and responding to ad hoc questions. In addition, a series of four events were held: Event 1 aimed to share information and explore experiences. Events 2 and 3 aimed to explore interim findings of the review. Event 4 disseminated findings. | Expert advisory group—academics, charity representatives, professionals. Four events: Event 1 (n = 15), parents, carers, professionals, researchers. Event 2 (n = 20), professionals. Event 3 (n = 25), parents and young people. Event 4 (n = 60), parents, professionals, policy makers. | Expert advisory group. Four events. | – | No information provided | No information provided |
Edwards et al. [43] | Individual interviews—aimed at identifying topics for review to focus on. Meeting—descriptive maps from initial scoping review/mapping exercise were presented. Informed by the principles of nominal group technique, participants generated independent lists of important topics, which were collated and displayed. Individual participants then ranked, in writing, their personal priorities from the list. | Individual interviews (n = 6), with young people (patients) or their carers. Face-to-face meeting—health professionals, young people, charity representative (n = 7), plus research team (n = 7) | Two methods: 1) Individual interviews 2) Face-to-face meeting | Consensus decision-making techniques: nominal group technique | No information provided | No information provided |
Harris et al. [25] | Five ‘cross-organisation’ events and seven ‘within-organisation’ events. Also email discussions and opportunistic contact with researchers. Recruitment to the advisory network took place throughout the review, and different individuals had different levels of involvement, and at different stages. Some members contributed on multiple occasions and others only on a single occasion. | ‘Advisory network’—salaried workers, health trainers, volunteer health champions and programme coordinators with expertise in using peer support, and people who had originally received support before going on to become a peer support worker. | Total of 12 meetings (approximately 240 face-to-face contacts with around 120 participants) Format—various, including use of notes, flip charts, audio recording, post-it notes | Participatory approaches | No information provided | No information provided |
Hayden et al. [44] | Workshop 1 = The goals of the stakeholder workshop were to discuss methods for evidence synthesis in general; to discuss the objectives and approach of our specific project; to refine definitions and priorities within the project; and to discuss key findings, key messages, and dissemination plans. Workshop 2 = Key messages and the data tables were discussed at a second stakeholder workshop. | Selected local stakeholders (n = 19) (professionals; no consumers involved) | Group meetings (2) (half day). Voting for prioritisation. Format—activities and small group discussions to engage attendees, to encourage discussion, and to reach consensus. | – | No information provided | No information provided |
Higginson et al. [45] | ‘Transparent expert consultations (TEC)’ involving consensus methods of nominal group and online voting, and stakeholder workshops. | Panel of experts in trials, quantitative, qualitative and mixed-method research, within and outside palliative care, patients/consumers, service providers, clinicians, commissioners, national policy makers and voluntary sector representatives | Transparent Expert Consultation (5 meetings). 3 workshops—2 with patients/consumers, 1 with clinicians/policy makers | Transparent Expert Consultation (incorporates NGT) | Ethics The research ethics committee of the University of Manchester (reference number 10328) approved the TEC component of MORECare. All TEC participants gave written consent. | No information provided |
Hyde et al. [34] | Three meetings at different stages in the review process. Meeting 1 took place during the protocol design stage and aimed at refining the scope of the review. Meeting 2 took place at the review preliminary findings stage, aimed at interpreting the results and planning the dissemination. Meeting 3 took place at the writing up stage, aimed at agreeing final results and planning how to share results. | Patients (n = 5), members of an established patient research user group (patients with musculoskeletal conditions) | Each face-to-face meeting was 3 h. Format of meetings were a mixture of presentations with discussion, and using ‘small group techniques’ | – | No information provided | No information provided |
Jamal et al. [35] | Meeting 1—aimed at informing developed of review questions. Discussion around key terms and perspectives of potential interventions. Researchers used data from meeting to generate topics. Meeting 2—aimed at prioritising topics. Discussion around list of topics generated from meeting 1, followed by voting for topics considered most important for systematic review. Five priority systematic reviews were then carried out by the researchers. | Young people (n = 13 for meeting 1, and n = 13 for meeting 2), from an existing young people’s public involvement in research group (the existing group had 25 members, who were recruited via advertisements. The group met monthly). | Face-to-face meetings Meetings were 1 h. Meetings involved short presentations and group discussion. Meetings were ‘supplemented with an online discussion forum’. Meeting 1 and online discussion forum generated list of health topics. Meeting 2 agreed a final list and voted individually to prioritise. | Consensus decision-making techniques: voting/ranking | No information provided | Members were not rewarded directly from researchers. However, they received £15 vouchers for their monthly participation (not specific to this research project), had food and transport provided and were eligible for an annual residential teambuilding activity. |
Liabo [46] | A participatory approach was used to involve a group of young people in all stages of a review. | Young people (total of 20 people across all meetings; 5 people only came to 1 meeting, others to multiple meetings; 2 came to all meetings). Target group was young people (up to age of 25) with experience of being in care. | 20 review meetings | ‘Participatory methods’ | The study was granted ethical approval by the Institute of Education ethics committee (application number FCH 62) in November 2007. | This study adapted the payment regulations operated by PAS, which was a fixed amount per hour, specified for meetings, training (half of meetings) and presentations (higher than meetings). |
Liu et al. [47] | User engagement was undertaken throughout our project through the inclusion of lay members on our Independent Project Steering Committee; in addition, we held two user conferences, one to launch the study and one to share and discuss our preliminary findings. | A total of 81 delegates attended the first conference, and 71 the second conference, from a wide variety of stakeholder organisations. | Independent Project Steering Committee Two meetings: 1. To obtain feedback on the research proposal and methods 2. To present and discuss preliminary findings | – | Ethical arrangements outlined in protocol, but unclear if formal ethical approval applied for or granted | No information provided |
Martin et al. [36] | A PPI (patient and public involvement) event was organised to facilitate members of the public in the East of England to talk about their views on population screening for dementia. The aim was to contextualise the findings of a systematic review for a British audience. | Purposive approach (50 invited, 36 attended) | Meeting (1 all day public event). Held during the final stages of the systematic review | ‘Quasi-focus group format’ | No information provided | In recognition of participants’ time and to mitigate the risk of participant dropout, a fixed monetary honorarium of £80 was offered and reminder calls were made in advance of the event. Costs for supportive care were provided to partners if they attended without their care recipients, and travel costs |
McConachie et al. [48] | A ‘multifaceted approach to consultation’, including (1) consultation with young people with ASD, in groups and by email; (2) survey of health professionals; (3) consultation with parents—3 meetings throughout review process; (4) multiple stakeholder discussion day about the preliminary conclusions of the review. | People with ASD—12 young people and 8 adults responded. Health professionals—838 survey respondents Parents—7 participated in one or more meetings Multiple stakeholder day—16 people | Methods included meetings, survey and emails. | Q-sort for rating agreement with outcomes | No information provided | Young people: ‘Each respondent was given a shopping voucher in acknowledgement of their contribution’. Parents: ‘Parents were given a financial acknowledgement in addition to travel expenses, to recognise their time and expertise at each attendance’. |
McCusker et al. [49] | A literature review on collaborative mental health care for depression was completed and used to guide discussion at an interactive workshop. The workshop was held as part of pre-conference activities at the June 2011 Canadian Conference on Collaborative Mental Health Care in Halifax, Nova Scotia. | We invited a spectrum of stakeholders to participate, aiming to have roughly equal representation of 4 groups: primary care providers, mental health providers, decision makers and consumers. | Workshop (n = 40; 9 members of planning committee and 31 invited stakeholders) Survey, to rate agreement with themes arising from workshop (n = 21/42, 50% response rate) | Thematic analysis of workshop discussion | The protocol was approved by the St Mary’s Hospital Research Ethics Committee. Participants provided oral but not written consent to participate. Permission was requested at the workshop for the proceedings to be audio-recorded and photographed, and for the participants’ names and affiliations to be listed in the final report. As one person did not agree to audio recordings, no recordings were made. | No information provided |
McGinn et al. [56] | A Delphi study among Canadian representatives of actual or potential EHR users to confirm the findings of the systematic review and to prioritise the key barriers and facilitating factors for EHR implementation in Canada. | 106 participants: 14 physicians, 30 healthcare professionals, 33 managers, 29 health information professionals (0 patient representatives due to low recruitment). | Delphi—3 rounds: 83 participants completed round 1, 69 round 2 and 63 round 3. | Delphi approach | Ethics approval for the study protocol was received from the Research Ethics Board of the Centre Hospitalier Universitaire de Québec (approved January 23, 2009; ethics number 5-08-12-06). | No information provided |
Morgan et al. [58] | As part of a partnership approach for a wider project relating to incentive mechanisms for smoking cessation in pregnancy and breastfeeding, 2 mother-and-baby groups were recruited and were co-applicants on the wider project. | Members of 2 existing groups (groups of around 12 people). Groups were mother and baby/toddler groups. | 15 meetings (during project, not just systematic review component). Members contributed to interpretation of systematic review findings by providing feedback on a number of vignettes of studies included in the evidence synthesis. | ‘Participatory’ | Although the service user collaborators were independent or local government representatives, rather than NHS groups, we considered it preferable to gain ethics committee approval before active engagement, particularly because the groups had not been involved in research before. | No information provided |
Oliver et al. [37] (‘correlational’ review) | A one-off workshop/meeting with members of an existing group. The workshop was scripted. Ground rules were established. Literature had been screened but not synthesised: involvement explored variables and pathways that had been examined in the literature, aiming to contribute to a theoretical framework to help synthesise study findings. | Young people, aged 12–17 years (approximately 12) | One meeting (2.5 h). Format: discussion, ranking of themes | Used methods for ranking of importance (stickers and ranking cards), but these did not directly inform decision-making | No information provided | No information provided |
Oliver et al. [37] (‘views’ review) | A one-off workshop/meeting with members of an existing group. The workshop was scripted. Ground rules were established. The synthesis of literature was almost complete; the aim of involvement was to provide a check on the credibility of the synthesis and develop review implications. | Young people, aged 12–17 years (approximately 12) | One meeting (2.5 h). Format: discussion, ranking of themes | As above | No information provided | No information provided |
Oosterkamp et al. [54] | RAND-e modified Delphi (2 rounds) Group (consensus) meeting The purpose of the involvement was to develop consensus on statements on the prevention of WSL during orthodontic treatment with fixed appliances. | The expert panel comprised 11 representatives, all graduated dentists, from research, education, orthodontics, cariology, general dentistry and advisory general dental practitioners from health insurance companies. | RAND-e modified Delphi procedure (involves considering patient vignettes). Consensus meeting | Delphi approach | No information provided | No information provided |
Pearson et al. [50] | Two meetings at different stages in the review process. Meeting 1 took place at the start of the review and aimed to ‘sharpen the focus of the review so that it would be relevant to those directly involved in the implementation’. Meeting 2 took place when a draft of the review was complete, aimed at getting feedback on this draft. | Professionals (n = 10)—primary and secondary school-level educational professionals and senior academics linked to the review | Two face-to-face meetings (length unclear) | – | No information provided | No information provided |
Pollock et al. [38] | Three meetings at different stages in the review process. Meeting 1 aimed to explore how the intervention could be categorised within the review. Meeting 2 aimed to explore and agree on specific strategies to update and amend the current Cochrane systematic review, based on the decisions made during meeting 1. Meeting 3 aimed to explore the perceived clinical implications of the findings. | Patients, carers, professionals (n = 13) | Three face-to-face meetings (half day meetings) Additional contact by email throughout review. | Nominal group technique | The project was approved by the Glasgow Caledonian University School of Health and Life Sciences ethics committee (Reference: HLS12/40). | We costed for the direct expenses associated with involvement (that is, travel, subsistence), but not for any funding to pay for the time of group members. |
Rees et al. [51] | An advisory group, which met 3 times during the project. The specific tasks for the group included: • Advising on the most appropriate terminology relevant to the subject area; • Identifying the literature, particularly unpublished reports; • Identifying and prioritising a priori outcomes for analysis; • Informing decisions the review team had to make at key stages of the review; • Helping to disseminate the work through incorporating its findings into members’ respective areas of work, and publicising the review to colleagues and associates. | Researchers/academics, policy specialists, voluntary sector workers, and a practitioner, representing a number of organisations (n = 7) | Three face-to-face meetings, of around 2 h. Format—meetings were ‘semi-informal’, involving presentations, discussions and explicit consensus methods (voting) | Consensus decision-making techniques: voting | No information provided | No information provided |
Saan et al. [55] (review 1) | ‘A partnership between the commissioner and researchers’ | Academic experts (n = 2) | ‘discussion’ Format unclear | – | No information provided | No information provided |
Saan et al. [55] (review 2) | Participants in an anonymous online Delphi process: ‘39 experts from the Netherlands (34 researchers and five policy advisors) were invited by email. They were contacted through the contact list of a recent expert meeting on self-management in the Netherlands and through the professional network of members of the research group.’ | Academic experts (29 invited: 20 completed first round of Delphi, 17 second round, 16 third round) | Anonymous online Delphi | Consensus decision-making techniques: Delphi | No information provided | No information provided |
Serrano-Anguilar et al. [39] | Three rounds of electronic Delphi. Round 1—an open questionnaire to explore the treatments used by patients and perceived health problems associated with their disease. Round 2—aimed at prioritising the health problems identified in the first round. Round 3—aimed at reaching final consensus. | Patients with DA (n = 53) | Electronic Delphi (3 rounds) | Consensus decision-making techniques: Delphi method | This study was approved by the Ethics Committee of the Canary Islands since its coordination was carried out by the Planning and Evaluation Unit of the Canary Islands Health Service | No information provided |
Smith et al. [52] | A ‘reference’ group was set up for the project. There were 3 face-to-face meetings at different stages in the review. Meeting 1 was aimed at sharing ideas and agreeing how the group would work. Meeting 2 discussed the scope and remit of the review. Meeting 3 was aimed at identifying key messages from the findings and planning dissemination. Additional communication between researchers and the reference group occurred by telephone conversations, email, project website and newsletters. | Service user and carer advocacy group members (n = 26) | Three face-to-face meetings (length unclear) | – | No information provided | No information provided |
Steward and Oliver [40] | The wider project aim was to ensure that policies and resources arising from the systematic review were relevant to parents, taking into account their experiences and views. Policy and resources were developed by 12 multi-disciplinary expert groups. Parents were included in these expert groups to share experiences and join discussions. | 22 parents, with experiences of newborn screening | Group meetings (3), over an 18-month period. | – | No information provided | Paid: • Parents’ travel expenses to and from meetings • For coffee at pre-meetings • Honoraria for parents equivalent to £17 per hour at each meeting, given in gift vouchers: this came to £75–£100 for each meeting • A fee to organisations that assisted with recruitment, in recognition of the support these organisations provided to parents. The fee was either £8 per hour of meeting attended by their members, or later, because this became too difficult to administer, a one-off £50 fee per parent recruited. |
Vale et al. [41] | Patient research partners had an initial face-to-face meeting, then contributed to a number of review activities, including ‘providing feedback on the detailed information folders, helping to trace contact details for trial investigators, learning about data management and analysis and contributing to regular project newsletter’ | (1) Reference group—professionals, patient/charity representatives, patient (n = 7) (2) Patient research partners—women who had received treatment for cervical cancer (n = 6) | ‘Patient research partners’—also contributing to another research project—around 6 meetings/year. Additional communication via email | – | No information provided | No information provided |