Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Considerations from the risk of bias perspective for updating Cochrane reviews

Systematic Reviews20154:136

DOI: 10.1186/s13643-015-0122-3

Received: 9 June 2015

Accepted: 24 September 2015

Published: 6 October 2015

Abstract

Authors of Cochrane reviews are expected to update their reviews every 2 years. The updating process helps to ensure that reviews are current and include recent evidence. However, the updating process is time-consuming for authors, particularly when Cochrane methods evolve and authors are required to revisit some of the originally included studies.

The Cochrane Collaboration's ‘Risk of bias’ tool is a mandatory component of Cochrane reviews, providing an assessment of the potential biases of included studies. The tool has been modified most recently in 2011, and the expectation is that new versions will continue to be produced and utilised in all Cochrane reviews. In this commentary we discuss, in the context of updating scenarios that are likely to be encountered, the potential options systematic review authors may have recourse to when the Cochrane Collaboration's ‘Risk of bias’ tool has been modified between the original review and its update. We recommend that authors who are updating reviews should revise their original assessments of included studies using the most recent version of the risk of bias tool. Despite the increased workload, use of the most recent version of the tool facilitates consistency of methods and reporting both across and within reviews, and ensures currency to the methodological rigour.

Keywords

Updating ‘Risk of bias’ Cochrane methods Systematic reviews

Background

The Cochrane Collaboration publishes systematic reviews in the Cochrane Database of Systematic Reviews (CDSR). Cochrane reviews, like non-Cochrane ones, are expected to be current; and when authors agree to take on and complete a Cochrane review, the expectation is that the review will be updated every 2 years. This update must include a new search for studies, and if new studies are identified, they must be screened and the relevant data extracted. A new synthesis including both previously identified studies and newly found studies is then published [1].

Updating systematic reviews is a challenge both for review authors and others involved with the publication process, such as peer reviewers and editors. Cochrane updates are given a new citation as long as they include a new recent search and a screening process for articles [2]. There is a substantial amount of time and resources required to complete any systematic review update. The novelty of the original publication no longer exists, and it is possible that the members of the authorship team no longer collaborate or some team members may have moved on to different areas of research. In some cases, a new research team takes over the review and carries out the update. Most of the research on updating reviews has focused on signals identifying the need to update [38] and reasons and methods for updating [9, 10]. Although authors agree to update reviews when they take on the project initially, approximately only 20 % of Cochrane reviews are updated every 2 years [11].

One of the challenges of updating a review is incorporating new methods. The field of scientific writing continues to evolve [12, 13]. The Cochrane Collaboration has 16 method groups, which specifically target and develop the methods used in the conduct and reporting of systematic reviews. In this commentary we discuss, in the context of updating scenarios that are likely to be encountered, the potential options systematic review authors may have recourse to when the Cochrane Collaboration’s ‘Risk of bias’ (RoB) tool has been modified between the original review and its update. We recommend that in order to keep a review current, authors need to not only incorporate new evidence but also utilise the most up-to-date methodological guidance.

Recent history of Cochrane risk of bias assessment for randomised trials

There have been significant changes to the Cochrane RoB tool in the past 10 years. The three most recent versions are described below and summarised in Table 1.
Table 1

A comparison of Cochrane risk of bias approaches since 2006

Risk of bias version

Biases assessed

Specific domains

Mandatory component in Review Manager software

2006

Recommendation was made to assess selection bias (allocation concealment), performance bias, attrition bias and detection bias for every study.

 

Selection bias (specifically allocation concealment) was incorporated into Review Manager and scored as ‘adequate (A), unclear (B), inadequate (C) or that allocation concealment was not used (D)’. Other biases were not mandatory and likely not consistently assessed. No justification of judgement was required.

2008

Development of a tool for assessing risk of bias. Four biases as above, with ‘other bias’ category for authors to consider additional biases. No limit on the number of additional biases authors can identify.

Sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting and other domains which authors could add.

Assessment of six domains within five biases mandatory. Judgement of ‘Yes’ (low risk), ‘No’ (high risk) or ‘Unclear’. Justification of judgement mandatory for each domain for each study.

2011

2008 tool modified but no new biases added.

As per 2008 version, except blinding divided into two domains, one related to blinding of participants and personnel, and the other related to outcome assessors.

Judgements changed to ‘High risk’, ‘Low risk’ or 'Unclear risk’. Clarification of which category of bias domain refers to and additional guidance for other risk of bias.

According to the Cochrane Handbook for Systematic Reviews of Interventions (referred to as the Handbook below) published in 2006, the recommendation was made to assess selection bias, performance bias, attrition bias and detection bias for every study [14], but only allocation concealment was a mandatory component of the RoB table in the Review Manager software used at that time. There was no requirement for a description of justification of any decisions and there was likely inconsistency in both the decision-making and the content of this reporting.

In the 2008 version of the Handbook, there was a major revision to the RoB tool [15]. This version of the tool included five bias categories and six mandatory components. There were options for considerations of outcome-specific components. The assessment of each component was done using ‘Yes’ , ‘No’ or ‘Unclear’ categories. For each judgement, the review author was required to include a justification based on text within the review and where possible, a direct quote.

In 2011, the Handbook was updated and further changes were made to the RoB tool [16]. To distinguish between performance bias affecting actual outcomes and detection bias in their inaccurate estimation, this version of the RoB tool included a recommendation to separate assessment of blinding into two components, one for study participants and intervention providers, and one for outcome assessors, respectively. Judgements were changed to ‘High risk’ , ‘Low risk’ or ‘Unclear risk’ , but the requirement to justify all judgements remained.

In 2008, the Cochrane Handbook suggested that all Cochrane reviews include a summary of findings (SoF) table [17]. The risk of bias assessment influences the reporting of the quality of evidence of specific outcomes in the SoF table. Previous research has demonstrated that only 9 % of updated Cochrane reviews have a change in conclusions compared to the original review [18]. Changes in the new risk of bias assessments leading to the potential changes of the content of the SoF tables could influence the overall conclusions. Further, robustness of meta-analytic estimates and exploration between study heterogeneity in estimates of effects is often explored in sensitivity analyses—repeat primary meta-analyses limited to subgroups of studies. Changes in study level risk of bias assessments have the potential to change estimates of effects in sensitivity analyses and explain heterogeneity that was previously unexplained.

Given the slow rate of updating, it is common for newly published review updates to require a full reassessment of studies included in earlier versions. For example, if the original version of a review was published in 2004 and updated in 2014, then all originally included studies would require reassessment using the 2011 RoB tool for methodological consistency. In 2014, over 400 reviews were updated in CDSR. Over 75 % of these updates were published as original reviews prior to January 2011, indicating that at some point, authors should have reassessed the previously included studies using a second, more current RoB tool [19].

The 2011 version of RoB improves the transparency of published reviews and the framework for assessing risk of bias [20]. Both the additional mandatory categories and judgements for decisions provide an uncomplicated method for authors to judge the impact of the risk of bias on the review results. Depending on when the previous version of a review was published, it is clearly a substantial amount of work for authors of updates to revisit all of the included studies in the previous publication, reassess their risk of bias and adjust the results accordingly. Though the application of the Cochrane RoB tool is required as per the Methodological Expectations of Cochrane Intervention Reviews [21], only 74 % of Cochrane review groups reported enforcing the application of the tool to all randomised controlled trials when updating a systematic review [22].

Discussion

Possible approaches

There are four different risk of bias scenarios to consider that reviewers may encounter when updating reviews. For our purposes, the decision-making has been broken down into whether studies were included in the original review, and subsequently, whether any additional eligible studies were found in the updated search. The different scenarios are presented with a rationale for, and implications of, each decision. The scenarios below apply to reviews for which the research questions and 'population-intervention-comparison-outcome', or PICO, framework remain essentially unchanged. When important changes to the research questions or framework are required (e.g. new important harms have been identified, new population and disease indications have emerged, comparator(s) has changed because another treatment(s) has now also diffused into current practice, etc.), a new protocol must be developed to undertake an entirely new review rather than update an existing one. Decisions at this time must be made whether to proceed with an update or to publish a new review with the new parameters.

1. Updating reviews where previous version(s) had no included studies

  1. a.

    No new eligible studies identified in the updated search

     
After complete searching and screening according to predetermined criteria, there are still some systematic reviews that have no included studies, commonly referred to as empty reviews [23, 24]. If no new studies are found in the search for the updated review, and the review remains empty, then there is obviously no need to apply a RoB tool.
  1. b.

    One or more eligible studies found in the updated search

     

If one or more new studies are judged to be eligible for inclusion in the review from a new recent search, the clear choice would be to use the current 2011 RoB tool. The authors would be required to do full data extraction of the newly included studies and as such, would also be expected to perform a full assessment of the risk of bias.

2. Updating reviews where previous version(s) had included studies

  1. a.

    No new eligible studies identified in the updated search

     
One could argue in this case that redoing RoB assessments of previously included studies is not necessary (Fig. 1). The 2011 Cochrane Handbook does not strongly recommend the reassessment of included studies when no new eligible studies are found [25]. However, the only solid rationalization for not re-evaluating previously included studies is a minimization of the work required. The fact that the review will be labelled, published and cited as updated implies that the methods used are current. The best approach is for the authors to reassess all the original studies using the current RoB tool. As described above, authors will receive a new citation for the update, even if there are no new studies included. The rationale is that updating of reviews is a holistic undertaking in which all aspects need to be current and up-to-date; not only searches but also methodological approaches to critical appraisal and synthesis of evidence. An argument may be made that empiric evidence does not exist demonstrating meaningful differences between the current and earlier versions of RoB tool. Nonetheless, revisions leading to the current tool have been made based on author and user input. The endorsement of the current tool by the Cochrane Collaboration is reason enough why all studies within a review should be assessed using the most recent tool.
Fig. 1

Options for assessing risk of bias when updating reviews with one or more original studies

However, there are important challenges in incorporating the most current risk of bias tool into an update of a review. Depending on the age of the review, it is possible that the authors have performed a risk of bias assessment on all studies in the previous version of the review using an older version of the tool. One could argue that older studies may be subject to a time lag bias as authors of older studies may have used acceptable methods at the time the study was conducted, but unfortunately, those studies would be graded as higher risk of bias based on current standards. One example of this is trial registration or protocol publication (or availability) to detect or clarify the absence of selective outcome reporting. To comply with the U.S. Food and Drug Administration Modernization Act (FDAMA) of 1997, clinicaltrials.gov was launched in 2000 as a trial registration platform [26]. Trial registration became an ICMJE requirement in 2005 [26] and was later mandated under the U.S. Food and Drug Administration Amendments Act (FDAAA) of 2007 [27]. Prior to 2005, there was no easy process for trial registration, and it was very rare for protocols to be published for individual trials. Therefore, older studies are much more likely to receive an ‘unclear’ RoB assessment of selective outcome reporting compared with trials published after 2007. Granted this may also be true for older included studies included in the first version of a review, but for updating, we are recommending authors reassess the reviews which involves additional work and may impact on the results. Furthermore, authors of older studies would report their findings according to the journal requirements which may have been different 10 or more years ago when the Consolidated Standards of Reporting Trials (CONSORT) statement was less widely adopted [28].

When writing Cochrane reviews, authors report that a RoB criterion is ‘unclear’ when the information is not available. One approach to dealing with the lack of clear reporting of information is to contact individual study authors. In principle, this approach seems logical and very useful. Unfortunately, there are two major limitations. First of all, it can be very difficult to track down author teams for individual studies. Researchers change institutions, research interests change and teams split up; as a result, study processes which may or may not contribute to RoB may be difficult to recall or report accurately. Secondly, there is evidence that contacting study authors may lead to overly positive answers. In a survey of 104 trialists, using direct questions about blinding with named categories of trial personnel, 43 % responded that the data analysts in their double-blind trials were blinded, and 19 % responded that the manuscript writers were blinded [29]. This is unlikely to be true, given that such procedures were reported in only 3 and 0 % of the corresponding published articles, and that they are very rarely described in other trial reports. Updating a review is unlikely to change judgements on a specific criterion which was judged to be ‘unclear’ , as it will be even more difficult to obtain information from authors because at least two more years will have passed since the included study was published.
  1. b.

    One or more studies found in the updated search

     

The consistency of methods and reporting within a review is an important issue. Review authors will likely choose to assess all the studies similarly, either using a previous RoB tool or the more recent version. However, as argued above, the best practice approach is for review authors to apply the 2011 RoB tool to all included studies, both those included in the previous review publication and the newly identified studies for inclusion to maintain review currency and associated face validity, regardless of the number in each category. The discussion above is relevant for an update with new studies, and the added concern of consistency within a review makes it very difficult to rationalise using an old tool for newly found studies. It is worth noting that if the search strategy is modified, there may be studies identified which should be included that were published prior to the original publication but not captured in the previous version of the review. The lack of reporting guidelines and difficulty contacting authors would be an issue for these newly identified but older studies, but they would still be included in the review and require a full assessment including risk of bias.

This commentary discusses an important methodological consideration associated with the updating of systematic reviews that is currently not being enforced universally as authors update their systematic reviews. We recommend that systematic review authors should not only update their review with new evidence but also ensure that the methods used are current. This is particularly relevant for the study risk of bias assessment using the Cochrane RoB tool which has gone several iterative revisions over the past 10 years.

Although we believe the recommendations are sound, there are some implications. The original version of a review would have described in the methods the techniques that would have been used for the RoB assessment. Theoretically, authors who use the newer RoB tools for an update would not be following their protocol. It would be very important for authors to document how the RoB assessment has changed. Cochrane authors are required to describe differences between protocol and review, and the use of a more recent RoB tool in an update should be clearly documented.

For a new review, in the process of writing the protocol, the review authors are expected to consider and clearly document how they will incorporate the risk of bias assessments into the results. These same decisions and documentation should be followed when performing an update. Authors should refer to the most recent version of the Cochrane Handbook for guidance and outline in the methods and any changes to the approach of incorporation of risk of bias assessment into the results. Authors should also be prepared for the possibility that results (including judgement about the reviewers’ confidence in the estimates of effects or quality of evidence) and conclusions may change by incorporating a newer RoB tool into the methods. Besides the actual shift in the magnitude and direction of the estimates of effects with new evidence, using a more current risk of bias tool could also potentially change the assessment of the quality of evidence, overall conclusions or both.

It is clear that using the most recent RoB tool for studies that were assessed in an earlier version of a review, substantially increases the workload. However, the improvements in the methods and the consistency across all Cochrane reviews are critical. As discussed above, authors receive full citation credit for an update, and therefore should meet any expectations of the publishing body.

Implementation of these recommendations will continue to be a challenge. There are new MECIR standards being developed for updating Cochrane reviews. Inclusion of our recommendations in the standards and in the Handbook will likely increase the chance of the recommendations being followed.

There will continue to be new developments and major revisions to the RoB tool used in Cochrane reviews. There is a new RoB tool recently launched for non-randomised studies [30]. New biases and new assessment tools are being developed for randomised controlled trials as well, and there will continue to be a need for authors of updates to consider the best approach. It is likely that other review components such as summary of findings tables, searching and statistical analysis will all continue to evolve and lead to additional work for authors to consider when updating reviews. These considerations can also potentially apply to other methodological aspects of evidence syntheses as corresponding guidance evolves—for example, approaches to meta-analysing sparse data, subgroup analyses and recommendations for assessment of publication bias.

Conclusion

Updating Cochrane reviews, and non-Cochrane reviews, remains a challenge for many reviewers. Nonetheless, using the newest version of the RoB tool for all included studies, even those studies included in the previous versions of the review, is the recommended approach. Authors and readers should be well aware of the challenges newer methods of risk of bias assessments present when updating Cochrane reviews.

Abbreviations

CDSR: 

Cochrane Database of Systematic Reviews

CONSORT: 

Consolidated Standards of Reporting Trials

RoB: 

‘Risk of bias’

SoF: 

'Summary of findings'

Declarations

Acknowledgements

We would like to thank David Moher and Lucy Turner for their helpful comments on this paper.

Alain D. Mayhew worked for the Cochrane Methods Bias Group from September 2013 to September 2015; the group was supported by the Canadian Institutes of Health Research (CIHR Funding Reference Number—CON-105529).

Monisha Kabir worked for the Cochrane Methods Bias Group during the summer of 2014; the group was supported by the Canadian Institutes of Health Research (CIHR Funding Reference Number—CON-105529). Her time was also supported by the University of Ottawa Co-op Programme. Mohammed T. Ansari received no external funding for his work.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Knowledge Synthesis Group, Clinical Epidemiology Program, Centre for Practice-Changing Research (CPCR), Ottawa Hospital Research Institute
(2)
Biology Programme, University of Ottawa
(3)
School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa

References

  1. The Cochrane Collaboration. Cochrane Review updates. http://community.cochrane.org/editorial-and-publishing-policy-resource/cochrane-review-updates. Accessed 4 Oct 2015.
  2. Citation of updates for full published Cochrane reviews: policy change. http://www.editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/Citation%20of%20updates_FINAL.pdf. Accessed 12 Aug 2015.
  3. Shojania K, Sampson M, Ansari M, Ji J, Garritty C, Rader T, et al. Updating systematic reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2007. p. 1–135.Google Scholar
  4. Shekelle P, Motala A, Johnsen B. Assessment of a method to detect signals for updating systematic reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2014.Google Scholar
  5. Shekelle PG, Newberry SJ, Wu H, Suttorp M, Motala A, Lim Y et al. Identifying signals for updating systematic reviews: a comparison of two methods. Rockville, MD: Agency for Healthcare Research and Quality; 2011.Google Scholar
  6. Ahmadzai N, Newberry SJ, Maglione M, Tsertsvadze A, Ansari M, Hempel S, et al. A surveillance system to assess the need for updating systematic reviews. Syst Rev. 2013;2:104. doi:10.1186/2046-4053-2-104.View ArticlePubMedPubMed CentralGoogle Scholar
  7. Newberry S, Ahmadzai N, Motala A, Tsertsvadze A, Maglione M, Ansari M, et al. Surveillance and identification of signals for updating systematic reviews: implementation and early experience; methods research report. Rockville, MD: Agency for Healthcare Research and Quality; 2013.Google Scholar
  8. Sutton AJ, Donegan S, Takwoingi Y, Garner P, Gamble C, Donald A. An encouraging assessment of methods to inform priorities for updating systematic reviews. J Clin Epidemiol. 2009;62(3):241–51. doi:10.1016/j.jclinepi.2008.04.005.View ArticlePubMedGoogle Scholar
  9. Moher D, Tsertsvadze A. Systematic reviews: when is an update an update? Lancet. 2006;367:881–3.View ArticlePubMedGoogle Scholar
  10. Moher D, Tsertsvadze A, Tricco A, Eccles M, Grimshaw J, Sampson M, et al. When and how to update systematic reviews (Review). Cochrane Database Syst Rev. 2008;(1). doi:10.1002/14651858.MR000023.pub3.
  11. Takwoingi Y, Hopewell S, Tovey D, Sutton A. A multicomponent decision tool for prioritising the updating of systematic reviews. BMJ. 2013;347:f7191.View ArticlePubMedGoogle Scholar
  12. Kearney MH. Hoping for a TREND toward PRISMA: the variety and value of research reporting guidelines. Res Nurs Health. 2014;37(2):85–7. doi:10.1002/nur.21591.View ArticlePubMedGoogle Scholar
  13. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e100097. doi:10.1371/journal.pmed.1000097.View ArticleGoogle Scholar
  14. Higgins JPT, Green S. Assessment of study quality. Cochrane handbook for systematic reviews of interventions 4.2.6 [updated September 2006]; Section 6. In: The Cochrane library, Issue 4. Chichester, UK: John Wiley & Sons, Ltd; 2006.Google Scholar
  15. Higgins JPT, Altman DG. Chapter 8: assessing risk of bias in included studies. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Chichester (UK): John Wiley and Sons; 2008.View ArticleGoogle Scholar
  16. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: assessing risk of bias in included studies. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration. 2011.Google Scholar
  17. Schünemann HJ, Oxman AD, Higgins JPT, Vist GE, Glasziou P, Guyatt GH. Chapter 11: presenting results and ‘Summary of findings’ tables. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Chichester (UK): John Wiley & Sons; 2008.Google Scholar
  18. French SD, McDonald S, McKenzie JE, Green SE. Investing in updating: how do conclusions change when Cochrane systematic reviews are updated? BMC Med Res Methodol. 2005;5:33.View ArticlePubMedPubMed CentralGoogle Scholar
  19. Archie—the Cochrane Collaboration database, search function. https://archie.cochrane.org. Accessed 13 Sept 2015.
  20. Savović J, Weeks L, Sterne JAC, Turner L, Altman DG, Moher D, et al. Evaluation of the Cochrane Collaboration’s tool for assessing the risk of bias in randomised trials: focus groups, online survey, proposed recommendations and their implementation. Syst Rev. 2014;3(1):37. doi:10.1186/2046-4053-3-37.
  21. Cochrane Editorial Unit. Methodological Expectations of Cochrane Intervention Reviews (MECIR). http://editorial-unit.cochrane.org/mecir. Accessed 18 Aug 2015.
  22. Reid EK, Tejani AM, Huan LN, Egan G, O’Sullivan C, Lawrence K. Managing the incidence of selective reporting: a survey of Cochrane review groups. 11th Annual Cochrane Canada Symposium, April 2014, Ottawa, Canada.Google Scholar
  23. Lang A, Edwards N, Fleiszer A. Empty systematic reviews: hidden perils and lessons learned. J Clin Epidemiol. 2007;60(6):595–7. doi:10.1016/j.clinepi.2007.01.005.View ArticlePubMedGoogle Scholar
  24. Yaffe J, Montgomery P, Hopewell S, Shepard LD. Empty reviews: a description and consideration of Cochrane systematic reviews with no included studies. PLoS One. 2012;7(5):e36626. doi:10.1371/journal.pone.0036626. Epub 2012 May 4.View ArticlePubMedPubMed CentralGoogle Scholar
  25. Higgins JPT, Green S, Scholten RJPM. Chapter 3: maintaining reviews: updates, amendments and feedback. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration. 2011.Google Scholar
  26. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med. 2004;141:477–8. Epub 2004 Sep 8. PMID: 15355883.View ArticlePubMedGoogle Scholar
  27. U.S. Public Law 110–85. Food and Drug Administration Amendments Act of 2007. 2007. p. 1–156.Google Scholar
  28. CONSORT Group. CONSORT: transparent reporting of trials. http://www.consort-statement.org/. Accessed 19 Aug 2015.
  29. Haahr MT, Hróbjartsson A. Who is blinded in randomised clinical trials? A study of 200 trials and a survey of authors. Clin Trials. 2006;3:360–5.PubMedGoogle Scholar
  30. Cochrane Bias Methods Group and Cochrane Non-Randomised Studies Methods Group. A Cochrane risk of bias assessment tool: for non-randomised studies of interventions (ACROBAT-NRSI). www.riskofbias.info. Accessed 4 Oct 2015.

Copyright

© Mayhew et al. 2015

Advertisement