Skip to main content

Living systematic reviews in rehabilitation science can improve evidence-based healthcare

Abstract

Although systematic reviews are considered as central components in evidence-based practice, they currently face an important challenge to keep up with the exponential publication rate of clinical trials. After initial publication, only a minority of the systematic reviews are updated, and it often takes multiple years before these results become accessible. Consequently, many systematic reviews are not up to date, thereby increasing the time-gap between research findings and clinical practice. A potential solution is offered by a living systematic reviews approach. These types of studies are characterized by a workflow of continuous updates which decreases the time it takes to disseminate new findings. Although living systematic reviews are specifically designed to continuously synthesize new evidence in rapidly emerging topics, they have also considerable potential in slower developing domains, such as rehabilitation science. In this commentary, we outline the rationale and required steps to transition a regular systematic review into a living systematic review. We also propose a workflow that is designed for rehabilitation science.

Peer Review reports

Background

Systematic reviews are considered to be the foundations of evidence-based practice [1,2,3]. The structured method of identifying, appraising, evaluating and synthesizing primary research findings facilitates clinical decision making based on the available evidence at a given time [4, 5]. Despite its widespread success, researchers have identified several problems in the conduct of systematic reviews. First, the average duration of performing a systematic review is almost a year from initial search to publication [1]. Furthermore, only a minority of the published reviews are updated and the median estimated time to update is approximately 3 years for Cochrane reviews and 5 years for non-Cochrane reviews [6, 7]. Given the continuously increasing flow of new trials, many systematic reviews become rapidly out of date [4, 8]. Second, poorly prepared systematic reviews are susceptible to bias, such as incomplete search strategies [9], errors in data extraction [10], and inter-rater disagreement in risk of bias assessments [11]. Although the effect on outcomes may be limited [12], the current system does not easily allow for corrections or updated practices after publication of the initial results.

Living systematic reviews (LSRs) are proposed as an alternative method to traditional systematic reviews that may offer a solution to these challenges. A systematic review is considered “living” when it includes a system by which it regularly incorporates newly available evidence into the analysis [13]. Generally, this system consists of a predefined cyclic workflow with planned updates at intervals ranging from days to months. Assisted by the availability of recent software developments that provide (semi-)assisted solutions for the review process, these updates take only a fraction of the time of conducting a baseline review [14, 15]. The chosen timeframe is at the discretion of the researchers and generally depends on the speed at which a topic of interest is expected to develop. Although LSRs have specifically been designed for rapid updating cycles of emerging topics, such as mapping the COVID-19 trials (COVID-NMA), protocols have been published that include updating cycles of 12 months [16,17,18].

The aim of this commentary is to illustrate the process of transitioning a regular review into a LSR and to propose a workflow for updating LSRs over time. We will use a recently performed research project where we transitioned a regular systematic review into a LSR as an example to discuss the required steps and key issues.

Main text

Characteristics of the baseline review

We performed the initial review in September 2020 [19] to identify the change in outcomes over time and the between-study heterogeneity for interdisciplinary pain treatment programs for patients with chronic pain. All longitudinal study designs such as case series and RCTs were considered. Our main reason for including more study types than RCTs was that carefully controlled inclusion criteria and treatment procedures do often not reflect the heterogeneous reality of clinical practice in the domain of rehabilitation [20,21,22,23]. In addition, many treatment programs disseminate their results through case series, which contain important information regarding the change over time. The abstract screening, full-text selection, data extraction, and risk of bias assessment was performed in duplicate. Primary studies were eligible if they included interdisciplinary multimodal pain treatment (IMPT) programs as intervention for patients with chronic primary pain that was primarily perceived in musculoskeletal structures. Furthermore, studies had to include a baseline and a follow-up assessment of at minimum 12 months post treatment for at least one of the outcomes of interest: Physical function, pain interference, self-efficacy, depression, anxiety, anger, general emotional function, social role functioning, and pain intensity. We assessed risk of bias with the JBI critical appraisal checklist for case series. Our data analysis included the change over time per outcome, by calculating standardized mean differences between pre-, post-, and final follow-up timepoints.

LSR protocol

A LSR starts with a baseline systematic review that is in line with the PRISMA reporting standard [13, 24]. In addition, there should be a study protocol that includes a rationale for maintaining a LSR as well as planned methodological and statistical approaches that are in line with multiple iterations over time [5, 13].

When the decision was made for the transition of this systematic review into a living systematic review, we submitted a new study protocol in PROSPERO (CRD42021247142). We chose to create a new record instead of updating the record of the baseline review, because the workflow and iterative nature of the living review are substantially different. The main rationale for this living systematic review was to provide and maintain an up-to-date overview of all interdisciplinary pain treatment programs that include a baseline and a follow-up measurement of at least 12 months post treatment on one of our outcomes of interest. Other reasons included a faster inclusion of primary studies in a meta-analysis, the possibility of accommodating new methodological developments into the study, and to update data extraction forms at a later stage, to include follow-up studies or correct omissions or errors.

Workflow

LSRs require long-term commitment of a research team, which can lead to a substantial ongoing workload [25]. Detailed planning of a feasible workflow that allows for updates of the procedures over time is therefore recommended [24, 26]. Researchers should anticipate that the research team is likely to change over time, and include training programs to calibrate assessment practices, and to ensure a correct interpretation of the screening and data extraction forms. This is also true for leaving the possibility open to integrate new digital developments to (semi-)assisted approaches within the review process at a later stage [13, 15].

Figure 1 depicts the workflow for the current project. Each year, we will rerun the search in all databases. New records will be imported into Endnote and deduplicated using the procedure of Bramer and colleagues [27]. The set of unique records will then be imported into web application Rayyan, for masked screening in duplicate [28]. Potential eligible studies will be imported into Endnote to help find the full text versions for these references. We will use Google Forms to perform the second-round selection, data extraction and risk of bias assessment. The advantage of these digital forms is that they are stored in an online spreadsheet that is connected to our digital application. Hence, new studies are automatically integrated into the online tables and calculations. The results of the update will be evaluated by the steering committee during the yearly meeting.

Fig. 1
figure 1

Workflow diagram of the living systematic review

The involved researchers are divided over two teams. Members of the steering committee will meet on a yearly base and decide whether new results justify a peer review publication, based on the number of new cohorts, changes in outcomes or conclusions. At minimum, we will publish an update every 3 years. This timeframe has been chosen to balance the prevention of redundant research efforts and scientific publications with the notion that an absence of any change also contains valuable information. The steering committee will also discuss potential methodology improvements or amendments to the procedures, including the opportunity to integrate large scale routinely collected data [14], the refinement of the data extraction form based on a newly developed checklist [29], and initiatives to increase participation of the research community [30]. Finally, the steering committee will evaluate the need for continuation of this LSR project based on the replication checklist by Tugwell and colleagues [31]. The meeting notes will be added as appendix to the online repository. Members of the review team will be responsible for screening, study selection, data extraction, and risk of bias assessment. They will receive training in all software applications and review procedures. For this purpose, we will create a training dataset consisting of 20 studies identified by the search of the baseline review, which will contain a mix of included and excluded studies. As recommended by Gagnier and colleagues [32], we aim to always include members with expertise in pain rehabilitation.

Dissemination

In addition to journal articles, LSR results are often disseminated through online applications [30, 33]. These digital platforms include all extracted data as well as interactive functions to visualize or summarize (a subset of) the data that is specifically relevant to the end-user [4, 30, 33]. Consequently, these interactive functions provide the opportunity to supply information to various stakeholders, including researchers, clinicians, policy makers, and patients.

To disseminate the results in an interactive way, we developed an R Shiny application that provides relevant and most up to date information of this review project [34]. The online platform contains key tables, time series, and forest plots. To optimize transparency, the raw data extraction forms and risk of bias assessments are also made available. Using functionalities of Shiny software, all plots and figures include interactive functions, including tables with search, filter and sort functions, forest plots with adjustable corrections for within-subject correlations and time series, with cohort selection and export functions. These functions help users to navigate to the available data, to explore differences and similarities between cohorts and to perform sensitivity analyses. The source code is available through GitHub, which also registers the amendments that will be made over time [35]. To further increase the relevance for clinicians and policymakers, the platform could be expanded in future iterations with additional modules, such as GRADE recommendations, that improve the description, interpretation, and understanding of the dataset (Table 1).

Table.1 Steps for transitioning a systematic review into a living systematic revie

Conclusions

Living systematic reviews provide a means to quickly integrate primary studies into meta-analyses, while upholding the scientific rigor of systematic reviews. Recent innovations in (semi-)assisted tools for conducting a systematic review have streamlined the procedures and have substantially reduced the efforts to update results over time. We believe that these up-to-date overviews, communicated through peer reviewed articles and online applications, positively impact the dissemination and implementation of scientific findings into clinical practice.

Availability of data and materials

The dataset generated by the living systematic review will be available in the impt-cohort repository: https://datascience.hu.nl/rsconnect/impt-cohorts/

References

  1. Chandler J, Hopewell S. Cochrane methods-twenty years experience in developing systematic review methods. Syst Rev. 2013;2(1):76.

    Article  Google Scholar 

  2. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Int Med. 2007;147(4):224–33.

    Article  Google Scholar 

  3. Stevens KR. Systematic reviews: the heart of evidence-based practice. AACN Advanced Crit Care. 2001;12(4):529–38.

    CAS  Google Scholar 

  4. Boutron I, Créquit P, Williams H, Meerpohl J, Craig JC, Ravaud P. Future of evidence ecosystem series: 1. Introduction—Evidence synthesis ecosystem needs dramatic change. J Clin Epidemiol. 2020.

  5. Simmonds M, Salanti G, McKenzie J, Elliott J, Agoritsas T, Hilton J, et al. Living systematic reviews: 3. Statistical methods for updating meta-analyses. J Clin Epidemiol. 2017;91:38–46.

    Article  Google Scholar 

  6. Bashir R, Surian D, Dunn AG. Time-to-update of systematic reviews relative to the availability of new evidence. Syst Rev. 2018;7(1):195.

    Article  Google Scholar 

  7. Rombey T, Lochner V, Puljak L, Könsgen N, Mathes T, Pieper D. Epidemiology and reporting characteristics of non-Cochrane updates of systematic reviews: A cross-sectional study. Res Synth Methods. 2020;11(3):471–83.

    Article  Google Scholar 

  8. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

    Article  Google Scholar 

  9. Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Errors in search strategies used in systematic reviews and their effects on information retrieval. J Med Lib Assoc. 2019;107(2):210.

    Google Scholar 

  10. Horton J, Vandermeer B, Hartling L, Tjosvold L, Klassen TP, Buscemi N. Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy. J Clin Epidemiol. 2010;63(3):289–98.

    Article  Google Scholar 

  11. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Seida JK, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. Bmj. 2009;339:b4012.

    Article  Google Scholar 

  12. Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol. 2005;58(7):741–2.

    Article  Google Scholar 

  13. Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, et al. Living systematic review: 1. Introduction—the why, what, when, and how. J Clin Epidemiol. 2017;91:23–30.

    Article  Google Scholar 

  14. Créquit P, Boutron I, Meerpohl J, Williams H, Craig J, Ravaud P. Future of evidence ecosystem series: 2. Current opportunities and need for better tools and methods. J Clin Epidemiol. 2020.

  15. Thomas J, Noel-Storr A, Marshall I, Wallace B, McDonald S, Mavergames C, et al. Living systematic reviews: 2. Combining human and machine effort. J Clin Epidemiol. 2017;91:31–7.

    Article  Google Scholar 

  16. De Zeeuw TI, Peters G, Baan A, de Regt L, Bolman C, Crutzen R. Extending the Earcheck intervention: behavior change to prevent hearing damage, with a focus on young people with low social-economic status 2020 Available from: https://osf.io/7xjhg/?branch = master.

    Google Scholar 

  17. Pollock A, van Wijck F. Cochrane overviews: how can we optimize their impact on evidence-based rehabilitation? Eur J Phys Rehabil Med. 2019;55:395–410.

    Article  Google Scholar 

  18. Winters M, Holden S, Vicenzino B, Welton NJ, Caldwell DM, Lura CB, et al. Which treatment is most effective for patients with patellofemoral pain? A protocol for a living systematic review including network meta-analysis. BMJ open. 2018;8(11).

  19. Elbers S, Wittink H, Konings S, Kaiser U, Kleijnen J, Pool J, et al. (accepted unedited manuscript). Longitudinal outcome evaluations of interdisciplinary multimodal pain treatment programs for patients with chronic primary musculoskeletal pain: a systematic review and meta-analysis. Eur J Pain.

  20. Morley S, Williams A, Eccleston C. Examining the evidence about psychological treatments for chronic pain: Time for a paradigm shift? Pain. 2013;154(10):1929–31.

    Article  Google Scholar 

  21. Rowbotham MC, Gilron I, Glazer C, Rice AS, Smith BH, Stewart WF, et al. Can pragmatic trials help us better understand chronic pain and improve treatment? Pain. 2013;154(5):643–6.

    Article  Google Scholar 

  22. Smeets RJ. How to sustain and improve client centred (matched) care in chronic musculoskeletal pain? Start by changing the way policy makers select and judge the large amount of available data, and get rid of the dogma of stepped care. Eur J Physiother. 2021;23(2):66–7.

    Article  Google Scholar 

  23. Wade DT, Smeets RJ, Verbunt JA. Research in rehabilitation medicine: methodological challenges. J Clin Epidemiol. 2010;63(7):699–704.

    Article  Google Scholar 

  24. Counotte MJ, Egli-Gany D, Riesen M, Abraha M, Porgo TV, Wang J, et al. Zika virus infection as a cause of congenital brain abnormalities and Guillain-Barré syndrome: from systematic review to living systematic review. F1000Research. 2018;7.

  25. Millard T, Synnot A, Elliott J, Green S, McDonald S, Turner T. Feasibility and acceptability of living systematic reviews: results from a mixed-methods evaluation. Syst Rev. 2019;8(1):325.

    Article  Google Scholar 

  26. Schmidt L, Olorisade BK, McGuinness LA, Thomas J, Higgins JP. Data extraction methods for systematic review (semi) automation: a living review protocol. F1000Research. 2020;9.

  27. Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. J Med Lib Assoc. 2016;104(3):240.

    Article  Google Scholar 

  28. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210.

    Article  Google Scholar 

  29. Negrini S, Armijo-Olivo S, Patrini M, Frontera WR, Heinemann AW, Machalicek W, et al. The randomized controlled trials rehabilitation checklist: methodology of development of a reporting guideline specific to rehabilitation. Am J Phys Med Rehabil. 2020;99(3):210–5.

    Article  Google Scholar 

  30. Probst P, Hüttner FJ, Meydan Ö, Kalkum E, Kretschmer R, Jensen K, et al. Evidence map of pancreatic surgery: protocol for a living systematic review and meta-analysis. BMJ Open. 2019;9(9):e032353.

    Article  Google Scholar 

  31. Tugwell P, Welch VA, Karunananthan S, Maxwell LJ, Akl EA, Avey MT, et al. When to replicate systematic reviews of interventions: consensus checklist. BMJ. 2020:370.

  32. Gagnier JJ, Morgenstern H, Altman DG, Berlin J, Chang S, McCulloch P, et al. Consensus-based recommendations for investigating clinical heterogeneity in systematic reviews. BMC Med Res Methodol. 2013;13(1):106.

    Article  Google Scholar 

  33. Matl S, Brosig R, Baust M, Navab N, Demirci S. Vascular image registration techniques: a living review. Med Image Anal. 2017;35:1–17.

    Article  Google Scholar 

  34. Elbers. Interactive dashboard: IMPT cohorts 2020 [Available from: https://datascience.hu.nl/rsconnect/impt-cohorts/.

  35. Elbers S. Impt meta_analysis, Github repository; 2021.

    Google Scholar 

Download references

Acknowledgements

Not applicable

Funding

Not applicable

Author information

Authors and Affiliations

Authors

Contributions

SE developed the initial idea and prepared the manuscript. All authors further discussed the idea and contributed to the final manuscript. HW, JP, and RS helped supervise the project. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to S. Elbers.

Ethics declarations

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Elbers, S., Wittink, H., Kaiser, U. et al. Living systematic reviews in rehabilitation science can improve evidence-based healthcare. Syst Rev 10, 309 (2021). https://doi.org/10.1186/s13643-021-01857-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-021-01857-5

Keywords