Summary of main results
The update of the SR Toolbox aims to provide a simple and easily navigable interface for researchers to discover guidance and software tools to help conduct systematic reviews and wider evidence syntheses projects. The new structure of the SR Toolbox, which incorporates the ability to search by review family and review stage, has been developed and implemented to make it easier for researchers and other stakeholders with less familiarity and experience with the underlying computational concepts of tools. Stakeholders should be more able to identify and access software and guidance that may assist them with their evidence syntheses projects.
Our brief analysis of tools included in the platform up to February 2022 suggests that many software tools and guidance documents currently within the SR Toolbox can potentially be applicable to multiple review families, though reviews of reviews and qualitative reviews may currently be less well served. Guidance documents largely focus on methods for critical appraisal, followed by reporting guidelines, with far fewer publications surrounding other aspects of the review production process. Additionally, software tools to support the systematic review process may be mostly well-maintained and up to date, though there is some uncertainty surrounding this. The trajectory of guidance and reporting frameworks for evidence syntheses being published has been steadily increasing and has seen a particular increase since 2015.
Strengths and limitations of this work
Well-defined categories were used to map the guidance and software tools, based on widely accepted published standards [5]. These categories were agreed upon by highly experienced systematic reviewers (EEJ and CM) and information specialists (HOK and AS). Two editors with considerable expertise in computational and data science (CM and HOK) were responsible for the construction of the updated SR Toolbox.
However, there are some limitations of this work. The initial mapping exercise was conducted by a single reviewer, with a second checking some records for accuracy. This may be considered a bias, as it is possible that there may some minor inaccuracies in coding and charting of the tools and guidelines.
Potential areas for future research
As part of the mapping exercise for this work, we added a column in our Excel sheet to identify when the software tool or guideline was added to the SR Toolbox. This will allow us to determine the trajectory of publications and the rate at which new software tools are being added in the future more accurately.
This column may be one way of identifying areas for expansion or refinement within future iterations of the SR Toolbox. For example, there may also be an argument to further refine the ‘Other’ category in the SR Toolbox in future updates, particularly to highlight software tools and guidance relating to network meta-analyses and prognostic reviews. A 2016 review identified 456 network meta-analyses including at least four interventions [10], suggesting that the review type is increasing in number. Prognostic reviews have been formally adopted by Cochrane, with the first two Cochrane prognostic reviews published in 2018 [11, 12], while there have also been calls for more prognostic reviews to be conducted in response to a growing amount of primary prognostic research [13].
Living systematic reviews have also been proposed as a contribution to evidence synthesis by providing high-quality reviews that are updated as new research in the area becomes available [14]. We discussed the inclusion of living systematic reviews as a standalone review category within the new iteration of the SR Toolbox, as there has been some evidence that machine learning has been used to support the production of these reviews [15], but currently the SR Toolbox does not contain any specific guidance or software tools relating to living systematic reviews. If software tools and guidelines become available for living systematic reviews, we will consider adding this review category to the Toolbox in the future.
More generally, the mapping exercise and subsequent analysis has highlighted some areas for further research and tool production. Tools and guidance to support reviews other than systematic reviews of intervention effectiveness may be needed, particularly for reviews of reviews and qualitative reviews. Additionally, there are also very few tools or guidelines relating to stakeholder engagement in the review production process. While general guidance on how to report patient and public involvement in research exists in the form of GRIPP2 [16], and the ACTIVE framework has been developed to describe stakeholder involvement in systematic reviews [17], there are currently few other frameworks or tools specifically designed to help researchers undertaking evidence syntheses to involve wider stakeholders in the process.