This study will follow the key steps recommended by the EQUATOR Network for developing reporting guidelines in health research [8], which have been used to successfully develop reporting guidelines for other similar knowledge syntheses such as systematic reviews [11] and network meta-analyses [26]. We will develop the reporting guideline in four stages: (1) project launch, (2) literature review, (3) modified Delphi exercise, and (4) development of the guidance statement. These stages are illustrated in Fig. 1 and described below. We will record protocol amendments and describe these in our final manuscript. We have registered our intent to develop the PRIOR reporting guideline for overviews on the EQUATOR Network website (“reporting guidelines under development” section) [27]. The EQUATOR Network encourages registration of all reporting guidelines and extensions under development to raise awareness and help to prevent duplication. Before launching the project, we obtained ethics approval from the University of Alberta Health Research Ethics Board (#Pro00086094).
Project launch
At project launch (December 2018), we established a core project team who will be responsible for the day-to-day operations of the planned project, consisting of methodologists with expertise in overview methodology and the development of reporting guidelines (MP [Canada]; LH [Canada]; RMF [Portugal]; DP [Germany]; ACT [Canada]), and research staff (AG and MG [Canada]).
We then established an international and multidisciplinary expert advisory board, with expertise in overview methodology and the development of reporting guidelines, who will advise on the development of PRIOR: Dr. David Moher (Canada); Dr. Tianjing Li (USA); Dr. Sue Brennan (Australia). The advisory board will be consulted regularly throughout the guideline development process. They will be asked to recommend relevant documents for the literature review; nominate participants for the Delphi exercise, review the checklist items for inclusion in the first round of the Delphi exercise, provide feedback after each round of the Delphi exercise (e.g., interpret results of the previous round, approve content for the next round), help plan and co-facilitate the in-person meeting, contribute to the production of the final reporting guideline, and assist with dissemination and knowledge translation activities.
Literature review
To support the development of the PRIOR reporting guideline, we will conduct a methodological SR examining the quality of reporting of a sample of overviews of healthcare interventions published from 2012 to 2016 (in progress) [28]. To ensure that the most recently available data are used to inform the PRIOR guideline, we have supplemented our own data with those recently published by Lunny et al. in 2019 [4]. We will also search for and summarize methodological documents related to conducting and reporting overviews (e.g., documents that provide guidance for reporting overviews, documents that summarize methods used to conduct overviews, and empirical studies that evaluate methods for conducting overviews) (in progress). To identify relevant methodological documents, we will use and expand upon the most relevant search strategies contained within the scoping review by Pollock et al. [7], the methodological SR by Pieper et al. [28], and the evidence map by Lunny et al. [5, 6]. Our searches, which will continue until the final version of the PRIOR guideline is completed, will consist of: web searches (Google Scholar); reference tracking [29, 30]; monitoring article alerts; hand-searching websites, conference proceedings, and personal files; and asking experts (e.g., advisory board members; Delphi participants during the first survey round) for relevant articles. Two independent reviewers will assess titles and abstracts and all potentially relevant full-text articles for inclusion, with discrepancies resolved by consensus or third party adjudication. Results of the literature review will be used to develop and refine a list of checklist items that a panel of expert participants will assess for inclusion in the PRIOR reporting guideline.
Modified Delphi exercise
A panel of experts (i.e., “participants”) will participate in a modified Delphi exercise to achieve a high level of agreement (≥ 70%) on the list of items to be included in the PRIOR reporting guideline [8]. The participants will provide feedback on potential items during three Delphi rounds, using structured feedback between rounds to help reconcile individual opinions and achieve group agreement [31,32,33]. Three rounds are likely to result in convergence of opinions between participants [32, 33]. The first two Delphi rounds will include all participants and occur online via self-administered survey; the third Delphi round will include a smaller subset of the participants (i.e., “expert panelists”) and occur during an in-person meeting that will use a nominal group technique (i.e., a formal consensus technique where expert panelists systematically review, discuss, and re-rate outstanding items) to achieve agreement [33, 34]. Should a high level of agreement (≥ 70%) not be reached following the in-person meeting, we may implement a third online survey among the expert panelists to assist in achieving agreement. Before the study, we will pilot test the survey’s usability, clarity, and face validity by sending it to five individuals familiar with overview methods but uninvolved in the current project. Their feedback will be used to revise the survey format and checklist items as needed.
Participant recruitment
We will use a purposive sampling technique [31] to identify and recruit a panel of up to 100 participants with experience coordinating, conducting, reviewing, disseminating, and/or using overviews of healthcare interventions (e.g., editors, authors, peer reviewers, and end-users of published overviews such as guideline developers). We will aim to recruit international participants who have diverse roles (e.g., researchers, healthcare professionals, patients, journal editors, policy-makers, funding agency representatives) and are employed in a range of settings (e.g., universities, hospitals, government, non-profit organizations, for-profit organizations). Participants will form a single panel for the analysis of survey results, feedback between rounds, and criteria for agreement. A list of potential participants will be prioritized by the core project team, in order to provide representation from our populations of interest (i.e., editors, authors, peer-reviewers, end-users) and major evidence synthesis centers (i.e., Joanna Briggs Institute, Canadian Agency for Drugs and Technologies in Health, Campbell Collaboration, Cochrane, Agency for Healthcare Research and Quality Effective Health Care Program, Canadian Task Force on Preventive Health Care, Centre for Evidence-Based Medicine, Centre for Evidence-based Health Care). The core project team and expert advisory board will also provide input, based on their knowledge of the potential participants, on each participant’s level of expertise and potential for unique contribution to the development of the guideline (e.g., statistical expertise, information specialists). Participants will be invited via personalized email that will describe the PRIOR guideline development project and explain the objective, process, and timelines of the Delphi exercise. We will obtain informed consent from participants using an online consent form.
Round one: Online survey
The participants will be asked to use a self-administered online survey to rate, on a four-point Likert scale, the extent to which they agree with the inclusion of each checklist item in the PRIOR reporting guideline (1 = strongly disagree, 2 = somewhat disagree, 3 = somewhat agree, 4 = strongly agree) [34]. Participants may also choose to answer “I don’t know” and provide an explanation [35]. For each item, a free text box will be provided for general comments (e.g., justification for their decision, proposed wording changes). Items will be presented in an order that reflects the progression of reporting in overviews (e.g., title, abstract, background, methods, results, discussion, other). At the end of the survey, two free text boxes will be provided for participants to suggest additional checklist items, and relevant methods papers. The Dillman principles for constructing respondent-friendly web surveys will be used to design the survey and its component items [36, 37]. Round one of the survey will remain open for 1 month (February–March 2020), during which bi-weekly reminder emails will be sent. The survey will be completed quasi-anonymously (i.e., the core project team but not the other study participants will know the identities and responses of the participants [31]), using a versatile online Delphi platform (e.g., Welphi). We will collate and summarize survey results, with agreement defined a priori as ≥ 70% for inclusion in (i.e., score of 3–4) or exclusion from (i.e., score of 1–2) the reporting guideline [38], based on the total number of responses obtained per question.
Round two: Online survey
In the second online survey (June–July 2020), the same participants will view and/or re-rate the checklist items presented in the first survey [32, 33]. The content, structure, and process will be similar to that of the first survey, with two differences. First, checklist items may be re-worded and/or re-formatted (e.g., items may be split or combined) based on the free-text comments collected in the first survey. Second, we will provide participants with their individual responses and a summary of anonymized group ratings, including all free-text comments, from the first survey. Participants will be asked to consider the structured feedback to inform their responses [32, 33]. Checklist items will be presented in the same order as previously. The items that reached agreement (≥ 70%) after the first survey will be presented for information purposes only (i.e. no more voting will occur, though participants may respond to free-text comments). The items that did not reach a high level of agreement during the first survey will be re-rated. The participants will be asked to determine whether and how they wish to modify their original answers in light of the group responses and comments; they may also provide feedback on free-text comments if desired. At the end of the survey, we will ask the participants to rate and provide comments for each new checklist item generated by participants during the first survey. We will collate and summarize survey results, with agreement defined as previously described (≥ 70%).
Round three: In-person meeting
With input from the advisory board, we will select and invite a minimum of 8 to 10 participants (i.e., “expert panelists”), ideally representing different stakeholder groups, to convene at a 1-day, in-person meeting (October 2020), where a nominal group technique and real-time voting will be used to systematically discuss and resolve outstanding disagreements [8, 33, 34]. If needed, a subset of these expert panelists may participate remotely using interactive software that allows for real-time screen sharing, audio discussion, and user comments (e.g., Adobe Connect, GoToMeeting). Moderators with expertise in overview methods and previous experience conducting consensus meetings for reporting guidelines will facilitate the meeting. We will audio record the meeting and take meeting minutes. The objective of the meeting will be to obtain final agreement on the list of items to be included in the PRIOR reporting guideline.
Prior to the meeting, members of the expert panel will receive a copy of their second-round survey results. We will begin the meeting by briefly summarizing the items reaching agreement for inclusion in, and exclusion from, the reporting guideline. No further voting will occur for these items, but outstanding free-text comments will be presented and resolved as needed; this could include rewording or reformatting of existing items. The bulk of the meeting will use a nominal group technique to obtain a high level of agreement (i.e., ≥ 70% agreement, as previously described) on those checklist items still lacking agreement [33, 34]. Each item will be reviewed sequentially, using a three-step process. First, facilitators will present the group survey results, all free-text comments, and all relevant methodological literature related to each item. This can help structure the interaction, provide a common starting point for participants, and promote evidence-based discussions about guideline content [33]. Second, the expert panel will discuss, debate, and aim to resolve discrepancies in a structured large-group discussion [33, 34]. Third, expert panelists will vote on the inclusion or exclusion of remaining items from the guideline.
The expert panel will be asked to re-rate the extent to which they agree with the inclusion of each checklist item. The content, structure, and process of the survey will be similar to the online surveys, with two changes: (1) the “I don’t know” option will no longer be available and (2) no free-text comments will be solicited. Expert panelists will complete the survey anonymously using a secure, online, live voting platform (e.g., Sli.do) that they will access using their personal electronic devices, with agreement defined as previously described (≥ 70%). Aggregate survey results will be automatically compiled by the software and presented to the group at the end of the survey. If agreement has not been reached on all items following the survey, we will ask the expert panelists to divide themselves into one small group per checklist item. Each small group will engage in unmoderated discussion to achieve final agreement, with rationale, for each outstanding item. Post-meeting discussions may continue over email or teleconference (i.e., including the expert panelists and facilitated by the core team) to achieve agreement, or we may implement an additional online survey with expert panelists, if needed. The meeting will conclude by discussing the strategy for producing, publishing, and disseminating the final guideline. We will discuss the inclusion of a flow diagram, development of an accompanying explanation and elaboration document, who will be involved in which activities, and publication and knowledge translation strategies [8].
Development of the guidance statement
A small writing group will iteratively draft the final version of the PRIOR guidance document based on the final decisions of the expert panel. The writing group will consist of the core project team, with an open invitation issued to the advisory board members and expert panel members. We will aim to provide clear, concise, and unambiguous wording for each PRIOR checklist item. The reporting guideline will be circulated among all advisory board members and expert panel members to obtain final input and approval prior to publication.