Methods and guidance on conducting, reporting, publishing, and appraising living systematic reviews: a scoping review

Abstract

Background and objective

The living systematic review (LSR) approach is based on ongoing surveillance of the literature and continual updating. Most currently available guidance documents address the conduct, reporting, publishing, and appraisal of systematic reviews (SRs), but are not suitable for LSRs per se and miss additional LSR-specific considerations. In this scoping review, we aim to systematically collate methodological guidance literature on how to conduct, report, publish, and appraise the quality of LSRs and identify current gaps in guidance.

Methods

A standard scoping review methodology was used. We searched MEDLINE (Ovid), EMBASE (Ovid), and The Cochrane Library on August 28, 2021. As for searching gray literature, we looked for existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses. The screening was conducted by two authors independently in Rayyan, and data extraction was done in duplicate using a pilot-tested data extraction form in Excel. Data was extracted according to four pre-defined categories for (i) conducting, (ii) reporting, (iii) publishing, and (iv) appraising LSRs. We mapped the findings by visualizing overview tables created in Microsoft Word.

Results

Of the 21 included papers, methodological guidance was found in 17 papers for conducting, in six papers for reporting, in 15 papers for publishing, and in two papers for appraising LSRs. Some of the identified key items for (i) conducting LSRs were identifying the rationale, screening tools, or re-revaluating inclusion criteria. Identified items of (ii) the original PRISMA checklist included reporting the registration and protocol, title, or synthesis methods. For (iii) publishing, there was guidance available on publication type and frequency or update trigger, and for (iv) appraising, guidance on the appropriate use of bias assessment or reporting funding of included studies was found. Our search revealed major evidence gaps, particularly for guidance on certain PRISMA items such as reporting results, discussion, support and funding, and availability of data and material of a LSR.

Conclusion

Important evidence gaps were identified for guidance on how to report in LSRs and appraise their quality. Our findings were applied to inform and prepare a PRISMA 2020 extension for LSR.

Introduction

Systematic reviews (SRs) are essential to provide evidence-based answers to clinical and public health-related questions. Due to the continuous publishing of relevant primary studies in some areas, it is important to keep these SRs up-to-date [1]. One could achieve that goal by adopting the living systematic review (LSR) approach, which is based on an ongoing surveillance of the literature and continual updating [2]. Regular searches ensure that the SR includes the latest available evidence and remains up-to-date [2]. Therefore, LSRs are most suitable for high-priority topics with substantial uncertainty and frequent publications. When continually updating a review, it is important to report changes to the methodology and the findings in transparent and traceable ways, which can be challenging.

Few guidance documents address the conduct, reporting, publishing, and appraisal of LSRs. The Living Evidence Network developed in 2019 the “Guidance for the production and publication of Cochrane living systematic reviews” [3]. However, this guidance lacks certain aspects of the LSR methodology, which have been shown to be important in the last years with the rising number of LSRs conducted. While the recent update of the “Preferred Reporting Items for Systematic reviews and Meta-Analyses” (PRISMA) can be used for reporting LSRs, the statement indicates there may be some additional considerations that need to be addressed [4]. Also, the AMSTAR 2—Assessing the Methodological Quality of Systematic Reviews—tool [5] which was developed for the critical appraisal of the quality of SRs, does not consider LSRs.

Therefore, it is of high interest to summarize the literature evaluating methods of conducting, reporting, publishing, and appraising LSRs, as well as any guidance on those methods. Scoping reviews are particularly useful in the context of emerging evidence and act as a precursor for other topic-related projects [6]. This scoping review is part of a larger project to develop an extension of the PRISMA 2020 statement for living systematic reviews.

Objective

The main objective is to systematically collate methodological literature on guidance on how to conduct, report, publish, and appraise the quality of LSRs and to systematically map how much and what kind of evidence is currently available.

Methods

A protocol elaborating on the detailed methodology of this scoping review was already published [7]. The main differences in methods between the protocol and this scoping review are displayed in the Supplementary Table 1.

Scoping review methodology

To achieve the objective, we conducted a scoping review to identify and evaluate existing evidence and map the availability of methods papers, evidence gaps, and associated primary research gaps [6]. We followed the standard scoping review methodology guidance of the Joanna Briggs Institute [6] and applied the following steps:

• a) Identification of the research question

• b) Identification of relevant studies

• c) Study selection

• d) Charting the data

• e) Collating, summarizing, and reporting of the results [8]

Moreover, we adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist (see Supplement Table 2) for transparent reporting of the results [9].

Eligibility criteria

We included articles that devoted at least one paragraph to discuss methods or conceptual approaches on how to conduct, report, publish, or appraise LSRs. Such articles were ideally methodological or concept papers describing methods for LSRs, guidance (e.g., handbooks) for undertaking LSRs, issued by organizations that conduct evidence syntheses, and commentaries or editorials that discuss methods for LSR.

We excluded from our search, LSRs themselves, LSR protocols, and non-LSR-specific papers.

Identification of relevant studies

We searched MEDLINE (Ovid), EMBASE (Ovid), and The Cochrane Library. All searches were completed on August 28, 2021, and we searched from database inception. The search strategy was initially developed by a researcher experienced in developing literature search strategies with support from an information specialist (LH), as part of a larger project to develop an extension of the PRISMA 2020 statement for LSRs [10, 11]. The strategy was peer-reviewed and updated by another information specialist (IM). Please see Box 1 of the Appendix for the complete search strategy.

As for searching the “gray literature,” we looked for existing guidelines and handbooks on LSRs from organizations that conduct evidence syntheses (e.g., Cochrane handbook, Living Evidence network, JBI) using the Lens.org website. Additionally, we conducted an ancestry search to identify relevant LSR handbooks and guidance documents from the reference list of published LSRs. We performed a descendency search, using certain seminal documents (e.g., papers defining LSRs and Cochrane guidance), and tracked their citations via Google Scholar.

Article selection

Two authors (from among CI, NS, EA) contributed to screening independently and in duplicate titles and abstracts. We used a web-based systematic review software Rayyan (RRID:SCR_017584) for the screening process. To ensure a consistent screening procedure and optimize agreement, we developed and used a detailed written instruction form. We then screened for full text assessing eligibility, based on our predefined eligibility criteria. Disagreements and conflicts were solved by consulting a third author.

Data extraction and presentation

Two review authors (from among CI, NS, VP, SW, EA) extracted and cataloged the data on LSR-specific methodological aspects into a standardized and pilot-tested data extraction form in Microsoft Excel (RRID:SCR_016137). We extracted the main article characteristics and LSR-specific guidance data according to our predefined categories on (i) conducting, (ii) reporting, (iii) publishing, and (iv) appraising LSRs. The identified evidence was mapped by visualizing overview tables created in Microsoft Word. The items of the conducting category are based on the standard process of conducting a systematic review from the Cochrane Handbook [12], including the intermediate steps from describing the rationale to evidence synthesis. The reporting category includes the 27 items of the original PRISMA 2020 checklist [4] to identify whether LSR-specific reporting guidance exists for each of these items. The items of the publishing category are partly based on standard Cochrane guidance for systematic reviews [12] and the experiences of LSR authors within this author team. The LSR appraisal category is based on the 16 questions from the AMSTAR 2 tool [5]. Even though we extracted and classified the data according to these categories, we considered that items from one category (e.g., conducting LSR) could have an impact on items from another category (e.g., publishing LSR) and might even overlap. The extracted study characteristics and category items are listed in Supplementary Table 3.

Results

We identified 4590 references, potentially relevant to our research question. After having removed 1171 duplicates, we screened 3436 records on title and abstract and excluded 3379 records that did not meet the pre-defined eligibility criteria. We screened the full text of the remaining 57 records and included 17 papers from the database search in the scoping review. We also searched for “gray literature” and identified 49 potential records, from which we included five papers in the scoping review. In total, 21 articles from both searches were included in the scoping review. The detailed selection process and results are reported in the PRISMA flow diagram (see Fig. 1) [4].

Fig. 1

Flowchart of the database search and gray literature

The evidence map

The 21 included papers provided data for 40 of our pre-defined LSR-specific items. Methodological guidance was found in 17 papers for conducting LSRs, in six papers for reporting LSRs, in 15 papers for publishing LSRs, and in two papers for appraising LSRs (see Tables 1 and 2).

Table 1 Evidence map on the four categories of LSR guidance

Table 2 Summary of the number of papers reporting on each category, reported pre-defined items/sub-items, and gaps of guidance evidence

LSR conducting guidance

From the 17 papers including guidance on conducting LSRs, we mapped and summarized the reported guidance for each of our pre-defined items and sub-items (see Table 3). We found evidence for all the pre-defined items on conducting and almost all the sub-items. A particular high frequency of papers, more than half of the 17 included papers, provided guidance on certain sub-items such as the rationale for conducting a LSR and the screening tool of the search. Between one and five papers presented guidance on other sub-items, including changing and re-evaluating the inclusion criteria, the search (frequency, database, and who), the data extraction (frequency, who, and how), the quality and bias assessment (frequency and how), the data synthesis with meta-analysis if applicable (frequency, who, and how), the frequency of the certainty of evidence assessment, authorship changes, ongoing method support, and funding. Also, we found that some papers established very broad guidance on several steps of conducting a LSR [1, 3, 13,14,15, 25, 29]. The remaining papers reported more specific guidance on certain particular steps of the LSR conduction process. We could not identify any evidence for guidance on two sub-items: who carries out the quality and bias assessment and the certainty of evidence assessment.

Table 3 Evidence table on identified guidance for conducting LSRs with a narrative summary of extracted data

LSR reporting guidance

From the six papers providing guidance on reporting LSRs, we mapped the available data for each of the PRISMA items and sub-items and summarized the identified guidance (see Table 4). We found guidance on 13 out of the 27 PRISMA items for reporting a LSR. We identified a higher frequency of papers, three out of the six, providing guidance for PRISMA item 24 on the registration and protocol. One or two papers provided guidance for PRISMA items one until eight, 11, 13, 16, and 25. We noted that one paper [3] included particularly elaborated guidance on some of the PRISMA items, and the remaining papers provided guidance on a particular PRISMA item.

Table 4 Evidence table on identified guidance for reporting LSRs with narrative summary of extracted data

We could not identify any guidance for the PRISMA items on reporting the methods, including data collection process (9), data items (10), effect measure (12), reporting bias assessment (14), and certainty assessment (15). Further, there was no guidance identified for the reporting of results, including study characteristics (17), presenting the risk of bias in studies (18), results of individual studies (19), results of synthesis (20), reporting bias (21), and certainty of evidence (22). No data was found on reporting the three items (23a, 23bc, and 23d) of the discussion, on the item reporting support and funding (25), and on the availability of data and material (27).

LSR publishing guidance

From the 15 papers including guidance on publishing LSRs, we mapped the available data for our pre-defined items and sub-items and summarized the identified guidance (see Table 5). We found guidance for all of the pre-defined items and all the sub-items. We identified a particular high frequency of papers, more than half of the 15 included papers, providing guidance on certain sub-items such as the publication type, publication frequency, update publication trigger, and time point for transitioning out of the living mode. A lower frequency of papers included guidance on the remaining sub-items. Also, we note that some papers provide very broad guidance on several aspects of publishing a LSR [3, 14, 19, 29]. The other remaining papers provided more specific guidance on particular steps of the LSR publication process.

Table 5 Evidence table on identified guidance for publishing LSRs with a narrative summary of extracted data

LSR appraisal guidance

From the two papers including guidance on LSRs appraisal, we mapped the available data for each AMSTAR 2 tool question and some additional items and summarized the identified guidance (see Table 6). We found guidance on appraising LSRs for four of the pre-defined items. Among the two included papers, both provided guidance on the use of appropriate risk of bias assessment techniques and funding of included studies reported. One of each provided guidance on the assessment of protocol and review differences, the ongoing search, searched study registries, and gray literature. Moreover, we noted that one paper included more elaborated guidance on several aspects of quality appraisal [3]. We found no data for guidance on the remaining 14 items and two sub-items.

Table 6 Evidence table on identified guidance for appraising LSRs with a narrative summary of extracted data

Discussion

To summarize the results, we included 21 articles from both search approaches in the scoping review. These papers included data for 40 of our pre-defined LSR-specific sub-items. Methodological guidance was found in 17 papers for conducting LSRs, in six papers for reporting LSRs, in 15 papers for publishing LSRs, and in two papers for appraising LSRs. We identified guidance on conducting LSRs for all of our pre-defined items of interest. Lacking evidence only exists for two sub-items on who carries out the quality and bias assessment and on the certainty of evidence assessment. Thus, we can state from our findings that there is enough guidance available in the literature on how to conduct a LSR and no major evidence gaps have been found.

We identified major evidence gaps in literature on guidance for reporting LSRs. There is lacking guidance for many of the PRISMA sub-items, such as reporting on the methods, the results, the discussion, reporting support and funding, and the availability of data and material. We did not find any evidence gaps in the literature for guidance on publishing LSRs. The identified papers included guidance for all of the pre-defined items and sub-items on publishing LSRs.

Regarding the literature on guidance for appraising the quality of LSRs, we can state that most of the important key items are lacking, indicating major evidence gaps. These include appraisal aspects on eligibility criteria, explaining study selection, assessments of data in duplicate, the list and description of included studies, funding sources and COI declarations reporting, assessing the heterogeneity of results, impact of risk of bias assessment on results, and use as well as handling of preprints.

This scoping review has certain limitations. The search was conducted in 2021 and within this 2-year gap, we could have failed to identify additional literature published since. We only focused on our four predefined categories of LSR methodological aspects, including conducting, reporting, publishing, and appraisal of LSRs. Even though these categories were drafted based on existing LSR methods handbooks, the PRISMA reporting checklist for SRs and the AMSTAR 2 tool for appraisal, a different author team may have chosen different categories or emphasized other LSR aspects. Moreover, we included quantitative guidance literature, rather than qualitative reviews or reports, as these would have sat outside the scope of this paper. The methodology of a scoping review itself includes some limitations as well. The scoping review is an approach to inform research and decision-making on existing evidence gaps and the availability of literature within a certain field of interest. The main purpose is to map, identify, and inform for future systematic reviews or other types of syntheses. Thus, the scope of a scoping review is often limited to presenting what kind of evidence exists, without further investigating and synthesizing the data of each included reference.

For the specific objective of our project, the scoping review approach has important strengths. We used a sensitive search strategy developed by an experienced researcher and information specialist. The article selection process, including the screening and data extraction that have been conducted independently and in duplicate, adds to the quality of the systematic approach. Also, the data extraction form was piloted before by the author group. We developed and published a detailed a priori protocol for this scoping review, which pre-defines our objective, the methods used, and the reporting of the review.

Our findings are of utmost importance, as they reveal important evidence gaps in methodological guidance on the reporting and quality appraisal of LSRs. We cannot provide any rational explanation as to why there is a lack of guidance for certain LSR-specific aspects, such as reporting and appraisal, and for other aspects, higher frequencies of guidance exist. We believe that the first obvious methodological question that authors need to address when a LSR becomes a relevant approach for their investigation, is how to conduct this novel review type. Thus, the need for LSR-specific guidance on conduct was probably acknowledged very early and researchers addressed this question in handbooks and guidance papers. Regarding the aspect of reporting or appraisal, guidance already exists for similar review types and the need for updating this literature is increasingly being acknowledged and addressed, for instance, in the PRISMA 2020 extensions for LSRs. The results of this scoping review will inform other authors, researchers, and decision-makers and show them what guidance literature is available or needs to be updated.

Conclusion

From this scoping review, we can conclude that there is some important evidence for guidance on LSRs available. In terms of the numbers of identified sources including guidance, there is a high frequency of guidance papers on conducting and publishing a LSR. However, we identified less guidance on the reporting of a LSR and the least guidance on the quality appraisal of LSRs.

When considering our results from the scoping review, there is a particular need to develop and publish more guidance on how to adequately report in LSRs. An updated LSR-specific guidance document on reporting can be highly relevant for LSR authors, reviewers, editors, and other stakeholders involved in the LSR process. The scoping review results on reporting guidance have been used as a precursor and have been applied to inform and prepare a project on developing a PRISMA 2020 checklist extension for LSRs. The findings on the categories other than the reporting LSRs could be used by further author teams to re-evaluate and update existing guidance on SRs. Hence, we identified major evidence gaps for guidance on LSR appraisal. The AMSTAR 2 tool, which is currently used to assess the quality of SRs is not updated yet for the use of LSRs. This could be considered for further research, since there is an emerging need to develop an AMSTAR2 tool extension for novel methodological approaches to evidence syntheses, such as LSRs. Data can be made available upon author request.