Skip to main content
Download PDF

A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2

Systematic Reviews volume 10, Article number: 273 (2021) Cite this article

Abstract

Background

AMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another assessment tool which was designed to evaluate the level of bias present within a systematic review. Our objective was to compare, contrast and establish both inter-rater reliability and usability of both tools as part of two overviews of systematic reviews. Strictly speaking, one tool assesses methodological quality (AMSTAR-2) and the other assesses risk of bias (ROBIS), but there is considerable overlap between the tools in terms of the signalling questions.

Methods

Three reviewers independently assessed 31 systematic reviews using both tools. The inter-rater reliability of all sub-sections using each instrument (AMSTAR-2 and ROBIS) was calculated using Gwet’s agreement coefficient (AC1 for unweighted analysis and AC2 for weighted analysis).

Results

Thirty-one systematic reviews were included. For AMSTAR-2, the median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher (> 0.61). For ROBIS, the median agreement for all questions was also 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher.

Conclusion

ROBIS is an effective tool for assessing risk of bias in systematic reviews and AMSTAR-2 is an effective tool at assessing quality. The median agreement between raters for both tools was identical (0.61). Reviews that included a meta-analysis were easier to rate with ROBIS; however, further developmental work could improve its use in reviews without a formal synthesis. AMSTAR-2 was more straightforward to use; however, more response options would be beneficial.

Peer Review reports

Background

Systematic reviews have become a fundamental part of evidence-based medicine; they are considered the highest form of evidence as they synthesise all available evidence on a given topic [1]. Many will also combine data to give an overall effect estimate using a meta-analysis. However, the quality and standard of reviews varies considerably. If this is not understood, or in some way established, the results of many reviews might be overstated. Quality assessment tools have been developed to assess such variation in standards.

One previously heavily cited tool is the Assessment of Multiple Systematic Reviews (AMSTAR) scale [2] which has been widely used since its development in 2007. This scale was shown to be both reliable and valid [3]. However, it came under criticism for some issues with its design. It was argued by Burda et al. [4] that AMSTAR was lacking in some key constructs, in particular, the confidence in the estimates of effect. It also lacks an item to assess subgroup and sensitivity analysis. Further criticisms include issues such as the inclusion of foreign language papers as “grey literature” and the idea that the items can often partially but not fully meet the criteria was highlighted. Also, each item was not weighted evenly and there is a lack of overall score, which became problematic when trying to compare scores. Thus, an upgraded version (AMSTAR-2) was developed in 2017. The new version promised to simplify the response categories, align the definition of research questions with the PICO (population, intervention, control group, outcome) framework, seek justification for the review authors’ selection of different study designs (randomised and non-randomised) and included numerical rating scales for inclusion in systematic reviews, seek reasons for exclusion of studies from the review, and determine whether the review authors had made a sufficiently detailed assessment of risk of bias for the included studies and whether risk of bias was considered adequately during statistical pooling and when interpreting the results [5].

A second novel assessment tool that has undergone rigorous development was published in 2016 (Risk of Bias in Systematic reviews [ROBIS [6]]). It aimed to provide a thorough and robust assessment of the level of bias within the systematic review.

Description of the assessment tools

Assessment of multiple systematic reviews (AMSTAR-2)

The main aim of the AMSTAR-2 is a tool to assess the methodological quality of the review. It is made up of 16 items in total and has simpler response categories than the original AMSTAR version. Some sections are considered by the authors to be critical domains, which can be used for determining an overall score (see Appendix, Table 12 for more information on the critical domains). AMSTAR-2 is intended for assessing effectiveness. The tool can also be applied to reviews of both randomised and non-randomised studies.

ROBIS tool

The main aim of the ROBIS tool is to evaluate the level of bias present within a systematic review. The tool is made up of three distinct phases. Firstly, there is an optional first phase to assess the applicability of the review to the research question of interest. The second phase is made up of 20-items within four main domains: study eligibility criteria, identification and selection of studies, data collection and study appraisal, synthesis and findings. This phase is to identify concerns about the review conduct. Each domain has signalling questions and ends with a judgement of concerns of each domain (low, high or unclear). There is also a third phase consisting of three signalling questions to enable an overall assessment of bias rating to be given. ROBIS has a wide application and is intended for assessing effectiveness, diagnostic test accuracy, prognosis and aetiology [6].

Previous research

Due to the novelty of both tools, there is limited available literature comparing them; however, some work has been recently published.

One review team [7, 8] compared all three tools (AMSTAR, AMSTAR-2 and ROBIS), applying them to reviews that reported both randomised and non-randomised trials. The inter-rater reliability between four raters’ across 30 systematic reviews was analysed. Minor differences were found between AMSTAR-2 and ROBIS in the assessment of systematic reviews including a mix of study type. On average, the inter-rater reliability (IRR) was higher for AMSTAR-2 compared to ROBIS. They assumed that scoring ROBIS would take more time in general, and it was always applied after AMSTAR-2, but in fact the mean time for scoring AMSTAR-2 was slightly higher than for ROBIS (18 vs. 16 min), with huge variation between the reviewers. They also reported that some signalling questions in ROBIS were judged to be very difficult to assess.

Aim

The overarching aim of our work is to add to the literature and make a further comparison of both assessment tools in two overviews of reviews. Our team had previously completed two overviews on complementary and alternative medicine (CAM) therapies for two hard-to-treat conditions. One overview evaluated systematic reviews of various CAM therapies for fibromyalgia (FM) [9], and the other evaluated systematic reviews of CAM therapies for infantile colic [10].

Objectives

Due to some of the challenges we had using both tools in our overview of reviews work, we planned a formal assessment of both tools by completing the following comparisons and evaluations:

  1. 1.

    To compare the content of the tools

  2. 2.

    To compare the percentage agreement (IRR)

  3. 3.

    To assess the useability/user experience of both tools.

Methods

Two overviews of reviews were conducted by our team [9, 10]. The first reviewed CAM for fibromyalgia and assessed the included reviews using both the original AMSTAR tool [2] and ROBIS [6]. This review was published in 2016, prior to the development and publication of AMSTAR-2 [5]. Here, we reported on 15 systematic reviews of CAM for fibromyalgia, published between 2003 and 2014 which assessed several CAM therapies. Eight of the reviews included a quantitative synthesis.

We subsequently completed a second overview of reviews of CAM for infantile colic published in 2019 [10]. Here, we used the new AMSTAR-2 tool alongside ROBIS. We reported on 16 systematic reviews of CAM for colic, published between 2011 and 2018. The reviews investigated several CAM therapies, 12 of which included a quantitative synthesis.

We later returned to the fibromyalgia review papers and reassessed them all using the AMSTAR-2 scale, for consistency. This results in a total comparison of 31 reviews. The reviewers were not strict about the order of ratings.

Assessment of methodological quality/bias of the included reviews

Three reviewers (RP, VL, PD) independently assessed each systematic review using both tools. Any reported meta-analyses were checked by a statistician experienced in meta-analyses (CP). The final score was agreed after discussion between the authors.

Data-analysis

Gwet’s AC statistic was used to calculate inter-rater reliability (IRR) [11]. Gwet’s AC2 is a weighted statistic which allows for “partial agreement” between ordinal categories. Therefore, Gwet’s AC2 was used to calculate IRR (using linear weights) for AMSTAR-2 questions with options “no”, “partial yes” and “yes” (questions 2, 4, 7, 8, 9). Gwet’s AC1 is an unweighted statistic which measures full agreement only. Gwet’s AC1 was used for all other AMSTAR-2 questions.

All signalling questions for ROBIS were analysed using Gwet’s AC2 with linear weights where “no”, “probably no”, “probably yes” and “yes” were recoded as 1–4. As mentioned above, Gwet’s AC2 is a weighted statistic which allows for “partial agreement” between ordinal categories. Ratings of “no information” were treated as missing. Gwet’s AC1 was used for ROBIS domains. Agreement for AMSTAR-2 and ROBIS was classified as “poor” (≤ 0.00), “slight” (0.01–0.20), “fair” (0.21–0.40), “moderate” (0.41–0.60), “substantial” (0.61–0.80), and “almost perfect” (0.81–1.00), following accepted criteria [12]. All analyses were completed using Stata 16 (StataCorp. 2019; Stata Statistical Software).

Results

Our first objective was to compare the content of the tools (see Table 1). Any overlaps and discrepancies between the two scales are identified. Overall, we found considerable overlap on the signalling questions. However, ROBIS does not assess whether there is a comprehensive list of studies (both included and excluded) or whether any conflicts of interest were declared (both at the individual trial level and for the reviews), as these are considered issues of methodology quality rather than bias. AMSTAR-2 also assessed possible conflicts of interest, which is not assessed in ROBIS, despite being a potential risk of bias. However, the section on synthesis was given more in-depth consideration in ROBIS tool.

Table 1 A comparison of the content of the two tools (AMSTAR-2 and ROBIS)
Full size table

Section 2: Comparison of the inter-rater reliability of the tools

AMSTAR-2

The consensus results for AMSTAR-2 of both fibromyalgia and colic overviews can be found in Table 2. We report on 15 systematic reviews of CAM for fibromyalgia and found all but one review [13] rated as having critically low confidence in the results (see Appendix, Table 15 for scoring information). This was the only Cochrane review included in the FM overview. We also report on 16 systematic reviews of CAM for colic. Most were rated as having critically low confidence in the results, 4 were rated as low and 1 (a Cochrane review) was considered to have high confidence in the results. The comparison of the ratings for each review can be found in the Appendix (see Tables 9, 10, 13, and 14). There were a greater number of discrepancies between the overall risk of bias and quality ratings in the fibromyalgia reviews. The overall risk of bias/quality ratings was more consistent in the colic reviews.

Table 2 Agreed results of AMSTAR-2 for fibromyalgia
Full size table

Results of inter-rater reliability analysis for AMSTAR-2

A summary of the inter-rater reliability [IRR] for AMSTAR-2 can be found in Table 3. Seven questions that relate to critical domains were identified by Shea et al. [5]; more information about these domains can be found in Appendix (Table 15).

Table 3 The inter-rater agreement between the three raters for AMSTAR-2
Full size table

Summary of the findings on Inter-rater reliability

In total, 460 comparisons were included in the analysis for AMSTAR-2. The median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher. There was almost perfect agreement for questions 2 (did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?), 7 (did the review authors provide a list of excluded studies and justify the exclusions?) and 10 (did the review authors report on the sources of funding for the studies included in the review?). The lowest agreement was for question 14 (did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?). Ratings were missing in 35 cases. The results are displayed in Fig. 1.

Fig. 1
figure 1

Gwet’s statistic for the inter-rater agreement for AMSTAR-2 questions

Full size image

The AMSTAR-2 critical questions, in particular, seemed to have good agreement compared to the other questions. There was at least substantial agreement for all critical questions except question 13 which had moderate agreement. Questions 2 and 7 both had almost perfect agreement and had the highest agreement of all AMSTAR-2 questions.

Gwet’s AC2 statistic was used for questions 2, 4, 7, 8 and 9. Gwet’s AC1 statistic was used for all other questions. The markers represent the Gwet’s statistic and the error bars represent the 95% confidence intervals. The italicised data represent the median value for all questions.

Further information on the separate reviews can be found in the Appendix (Tables 7 and 11). The overall median IRR agreement for AMSTAR-2 questions for fibromyalgia is 0.65 and for colic is 0.60.

ROBIS

Summary of the ROBIS results

The consensus results for ROBIS for both fibromyalgia and colic overviews can be found in Table 4. With regard to the ROBIS results, domain 1 (which assessed any concerns regarding specification of study eligibility criteria), 9 fibromyalgia reviews achieved a low risk of bias rating overall and 6 colic reviews achieved a low risk of bias rating overall. In domain 2 (which assessed concerns regarding methods used to identify and/or select studies), 7 fibromyalgia reviews achieved a low risk of bias rating overall and 6 colic reviews achieved a low risk of bias rating overall.

Table 4 Tabular presentation for agreement of ROBIS results
Full size table

Domain 3 assessed concerns regarding methods used to collect data and appraise studies; 7 fibromyalgia studies and 10 colic reviews achieved a low risk of bias rating overall.

With regard to domain 4 (which assessed concerns regarding the synthesis and findings), more variation in the fibromyalgia scores was found, whereas most colic reviews were rated as high risk of bias in this domain. The reviews that did not conduct a meta-analysis were hard to assess using ROBIS.

The final section provides a rating for the overall risk of bias of the reviews; 7 fibromyalgia reviews achieved a low rating; 6, a high rating; and 2, were rated as unclear. Four colic reviews achieved a low rating; 4, an unclear rating; and 8, a high rating.

Results of inter-rater reliability analysis for ROBIS

A summary of the inter-rater reliability for ROBIS can be found in Table 5.

Table 5 Inter-rater agreement
Full size table

Summary of the findings on Inter-rater reliability

For ROBIS, there were 734 comparisons considered for the 24 questions. The median agreement for all questions was 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher. Ratings were missing in 9 cases. At least one rater said “no information” in 159 comparisons. Rater 1 used “no information” 73 times; rater 2, 50 times; and rater 3, 93 times. In 107 comparisons only one rater said “no information” and the raters all agreed only in 10 comparisons. “No information” was used most frequently for question 1.1 (did the review adhere to pre-defined objectives and eligibility criteria? 23 studies), question 4.2 (were all pre-defined analyses reported or departures explained? 22 studies) and question 4.5 (were the findings robust, e.g., as demonstrated through funnel plot or sensitivity analyses? 16 studies). The agreement was “moderate” for domains 1 (0.45) and 3 (0.36) and for the overall risk of bias (0.45). The agreement for domains 2 and 4 were “fair” (0.36) and “slight” (0.17), respectively. The results are summarised in Fig. 2.

Fig. 2
figure 2

Gwet’s statistic for the inter-rater agreement for ROBIS questions and domains

Full size image

Gwet’s AC2 statistic was used for the ROBIS questions (filled markers) and Gwet’s AC1 statistic was used for the ROBIS domains (hollow markers). The error bars represent the 95% confidence intervals. The italicised data represent the median value for all ROBIS questions.

Further information on the separate reviews can be found in the appendix (Tables 8 and 12). The median IRR agreement for all ROBIS questions for FM is 0.55 and for colic is 0.63.

Section 3: Usability of the tools

All three raters felt AMSTAR-2 was more straightforward and user-friendly than ROBIS. This might be because it does not require expertise in systematic reviewing to complete this tool, just knowledge of trial design.

Several issues arose from using the ROBIS tool as it required more consideration to complete. Within each domain, each question had five possible responses (yes, probably yes, probably no, no, no information), although at times it was difficult to distinguish between yes/probably yes and no/probably no. It also might be more helpful to have a choice of “no concerns/minor concerns/ major concerns/considerable concerns”, instead of “low/high/unclear” judgements that are currently at the end of each domain when assessing the overall judgement of concerns. Although there were perceived differences in the individual answers to each signalling question between reviewers, the overall rating of the domains was more consistent. Overall, domains 1–3 were easier to follow and score.

The most difficult domain to score was domain 4 which covers “synthesis of evidence”. This was reflected in the lowest agreement between raters (0.17). We found that this domain is currently better designed for a review with a meta-analysis, rather than a narrative synthesis. The guidance document that accompanies the tool is long and difficult to navigate. On the plus side, despite having subjective opinions (within each domain there was variation between the reviewers’ responses to the signalling questions), you can still end with a moderately consistent overall result (0.45).

The ROBIS tool provides an overall sense of risk of bias of the review. There is better coverage overall than AMSTAR-2 and more precision with the use of a final rating. From our observations only, higher quality reviews were quicker to appraise. In our analysis, the “no information” rating for ROBIS questions was treated as missing. The raters rarely agreed on when to use this rating. In most cases, when one rater reported “no information” for a ROBIS question, the other two raters gave a different rating.

Several issues arose from using AMSTAR-2. Sometimes, the raters would have opted for a “partially yes” option when only a binary option (yes/no) was available (Q13, Q14, Q16). Also, some questions were ambiguous; in particular, Q3 asks if authors explain their selection of study design (e.g., use of RCTs/non RCTs); some reviews merely report they included RCTs rather than justifying their selection, which caused discrepancies between raters.

Also, some questions might elicit a different response depending on the outcome, e.g., Q13 (whether risk of bias was discussed/interpreted within the results), which may vary depending on whether there were multiple outcomes, and thus, which outcome is being referred to.

The raters also felt it would be helpful to have a formal space to add comments to justify their decision to help with discussions, as in the more ambiguous reviews; decisions were more open to interpretation. ROBIS, on the other hand, has a large section where the reviewer is expected to add selected text to support their decision.

Regarding completion timings, we were able to establish how long it took to complete both tools for one of the overviews (colic). There was little difference in timings between rater 1 and 2 to complete both tools; in fact, it took rater 2 slightly longer to complete AMSTAR-2 than ROBIS which is surprising, considering the issues reported above. However, rater 3 took considerably longer to complete ROBIS than AMSTAR-2 (see Table 6).

Table 6 Mean (SD) completion time (in minutes) for colic paper
Full size table

Rater 3 was the most experienced reviewer and helped develop the ROBIS tool. They spent longer on bringing the evidence forward from the individual reviews into the ROBIS extraction form as recommended by the guidance document, whereas the other two raters only wrote cursory notes.

It is important to highlight that it is advised in the ROBIS guidance document that it is a tool aimed at experienced systematic reviewers and methodologists. We would agree with this recommendation but recognise that this is not often the case in many groups undertaking reviews.

Discussion

Summary of findings

The median inter-rater reliability (IRR) agreement for both AMSTAR-2 and ROBIS questions was substantial: 50% of AMSTAR-2 questions and 46% of ROBIS questions had substantial agreement or higher. For AMSTAR-2, 460 comparisons were included in the analysis. The median agreement for all questions was 0.61. For ROBIS, there were 734 comparisons considered for the 24 questions. The median agreement for all questions was also 0.61. It is interesting that the median IRR agreement for both tools was 0.61, demonstrating a similar level of rating between the two scales.

Results were similar when conducting the analysis for fibromyalgia and colic reviews separately (see appendix for independent overview results). For fibromyalgia, the median IRR value was 0.66 for the AMSTAR-2 questions compared to 0.56 for the ROBIS questions. For the colic studies both AMSTAR-2 and ROBIS had a similar median (0.60 for AMSTAR-2 and 0.63 for ROBIS).

It must also be considered that the ROBIS questions include more categories than most of the AMSTAR-2 questions. Most AMSTAR-2 questions are binary. Inter-rater agreement tends to be lower when there are more categories, as there are more possibilities for disagreement. Similarly, ROBIS includes more questions than AMSTAR-2 which can also result in more disagreement. However, despite these differences, the median agreement was the same for the AMSTAR-2 and ROBIS questions.

Usability of the tools

Several issues arose when using the ROBIS tool as it required more consideration to complete, which could become problematic in a large review. All three raters felt AMSTAR-2 was more straightforward and user-friendly than ROBIS. This might be because it does not require expertise in systematic reviewing to complete this tool, just knowledge of trial design.

AMSTAR-2 was considered quicker to work through than ROBIS, yet the median timings demonstrated only a slight increase in timing on AMSTAR-2 than ROBIS in two raters, although one rater did take considerably longer on ROBIS than AMSTAR-2. All raters felt domain 4 of ROBIS was particularly difficult to complete if there was no meta-analysis. Domain 4 would benefit from further development in order to assess reviews without a meta-analysis, as in some ways it is biassed against these types of reviews.

Relationship to background research

Previous research [7, 8] compared four raters’ assessments across 30 systematic reviews. They calculated the IRR using the Fleiss’ k [45]. The IRR for scoring the overall confidence in the SRs with AMSTAR-2 was fair (AMSTAR-2: κ = 0.30; 95% [confidence interval] CI, 0.17 to 0.43). The overall domain in ROBIS was fair (ROBIS: κ = 0.28; 95% CI, 0.13 to 0.42). Interestingly, for the overall rating, AMSTAR-2 showed a high concordance with ROBIS and a lower concordance with AMSTAR.

We were unable to directly compare our results against Pieper’s work, as the Fleiss’ kappa ignores the order of the categories (when there are more than two categories), which is why we used Gwet’s as it takes the order into account and allows for “partial agreement”. Also, Gwet scores tend to be higher than Fleiss scores in general, which makes comparisons difficult to conduct.

In Pieper et al.’s [7] study, ROBIS was always applied after AMSTAR-2, and the mean time for scoring AMSTAR-2 was slightly higher than for ROBIS (18 vs. 16 min), with huge variation between the reviewers, whereas in our study, the overall mean time (calculated for colic reviews only) was slightly higher for ROBIS than for AMSTAR-2 (24.4 min compared to 14.3 min), although the mean ROBIS result was largely influenced by one rater.

Potential bias in the overview process

One author evaluated their own work using AMSTAR-2 and ROBIS (RP: [19, 29]), although this work was also independently assessed by two other reviewers (VL, PD). In addition, one of the developers of ROBIS (PD) applied the ROBIS tool to assess the included reviews.

We had not planned to complete an IRR assessment of the two scales whilst completing these two overviews of reviews; therefore, we did not apply strict criteria to our assessment schedule, i.e., we did not apply the tools in any particular order. We also did not complete timings for some of our assessments in a systematic way.

Another issue is we compared our ratings over time, i.e., a batch of five papers were discussed before the next batch was assessed; this is likely to have led to greater consistency between the raters over time, but our numbers were too small to check this.

Conclusion

In terms of quality assessment, ROBIS is an effective tool for assessing risk of bias in a systematic review but is more difficult to use compared to AMSTAR-2. It is more complex to work through, which might be problematic in a large review. As suggested by the developers of ROBIS; it is best used by experienced systematic reviewers/methodologists. Reviews that included a meta-analysis were easier to rate, however, further developmental work could improve its use in systematic reviews without a meta-analysis. AMSTAR-2 was more user-friendly and was effective at measuring quality of a review but was a less sophisticated tool. Both tools could do with minor changes to help improve their useability for people conducting systematic reviews.

Availability of data and materials

Not relevant.

Abbreviations

CAM:

Complementary and alternative medicine

AMSTAR:

Assessment of Multiple Systematic Reviews

PICO:

Population, intervention, control group, outcome

ROBIS:

Risk of Bias in Systematic reviews

IRR:

Inter-rater reliability

References

  1. Murad H, Asi N, Alsawas M. New evidence pyramid. BMJ Evid Based Med. 2016;21:125–7.

    Article  Google Scholar 

  2. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;15(7):10.

    Article  Google Scholar 

  3. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62:1013–20.

    Article  PubMed  Google Scholar 

  4. Burda BU, Holmer HK, Norris SL. Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement. Syst Rev. 2016;5:58.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS group. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

  7. Pieper D, Puljak L, González-Lorenzo M, Minozzi S. Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. J Clin Epidemiol. 2019;108:26–33.

    Article  PubMed  Google Scholar 

  8. Lorenz RC, Matthias K, Pieper D, Wegewitz U, Morche J, Nocon M, et al. A psychometric study found AMSTAR-2 to be a valid and moderately reliable appraisal tool. J Clin Epidemiol. 2019;114:133–40.

    Article  PubMed  Google Scholar 

  9. Perry R, Leach V, Davies P, Penfold C, Ness A, Churchill R. An overview of systematic reviews of complementary and alternative therapies for fibromyalgia using both AMSTAR and ROBIS as quality assessment tools. Syst Rev. 2017;6(1):97.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Perry R, Leach V, Penfold C, Davies P. An overview of systematic reviews of complementary and alternative therapies for infantile colic. Syst Rev. 2019;8(1):271.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Gwet KL. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. 4th ed. Gaithersburg: Advanced Analytics, LLC; 2014.

    Google Scholar 

  12. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.

    Article  PubMed  CAS  Google Scholar 

  13. Deare JC, Zheng Z, Xue CC, Liu JP, Shang J, Scott SW, et al. Acupuncture for treating fibromyalgia. Cochrane Database Syst Rev. 2013;5:CD007070.

    Google Scholar 

  14. Holdcraft LC, Assefi N, Buchwald D. Complementary and alternative medicine in fibromyalgia and related syndromes. Best Pract Res Clin Rheumatol. 2003;17(4):667–83.

    Article  PubMed  Google Scholar 

  15. Baronowsky J, Klose P, Musial F, Haeuser W, Dobos G, Langhorst J. Qualitative systematic review of randomised controlled trials on complementary and alternative medicine treatments in fibromyalgia. Rhuematol Int. 2009;30:1–21.

    Article  Google Scholar 

  16. Terhorst L, Schneider MJ, Kim KH, Goozdich LM, Stilley CS. Complementary and alternative medicine in the treatment of pain in fibromyalgia: a systematic review of randomized controlled trials. J Manipulative Physiol Ther. 2011;34(7):483–96.

    Article  PubMed  Google Scholar 

  17. Terhorst L, Schneider M. Complementary and alternative medicine in the treatment of pain in fibromyalgia: a systematic review of randomized controlled trials. Portland: From International Research Congress on Integrative Medicine and Health; 2012. 15-18 May 2012

    Google Scholar 

  18. De Silva V, El-Metwally A, Ernst E, Lewith G, Macfarlane GJ. Evidence for the efficacy of complementary and alternative medicines in the management of fibromyalgia: a systematic review. Rheumatology. 2010;49(6):1063–8.

    Article  PubMed  Google Scholar 

  19. Perry R, Terry R, Ernst E. A systematic review of homoeopathy for the treatment of fibromyalgia. Clin Rheumatol. 2010;29(5):457–64.

    Article  PubMed  Google Scholar 

  20. Boehm K, Raak C, Cramer H, Lauche R, Ostermann T. Homeopathy in the treatment of fibromyalgia--a comprehensive literature-review and meta-analysis. Complement Ther Med. 2014;22(4):731–42.

    Article  PubMed  Google Scholar 

  21. Ernst E. Chiropractic treatment for fibromyalgia: a systematic review. Clin Rheumatol. 2009;28(10):1175–8.

    Article  PubMed  Google Scholar 

  22. Mayhew E, Ernst E. Acupuncture for fibromyalgia--a systematic review of randomized clinical trials. Rheumatology (Oxford). 2007;46(5):801–4.

    Article  PubMed  CAS  Google Scholar 

  23. Daya S. The efficacy of acupuncture in the treatment of fibromyalgia syndrome. J Acupunct Assoc Chartered Physiother. 2007;(3):35–46.

  24. Langhorst J, Häuser W, Bernardy K, Lucius H, Settan M, Winkelmann A, et al. Komplementäre und alternative Verfahren beim Fibromyalgiesyndrom. Systematische Übersicht, Metaanalyse und Leitlinie. Schwerpunkt. Schmerz. 2012;26:311–7.

    Article  PubMed  CAS  Google Scholar 

  25. Martin-Sanchez E, Torralba E, Díaz-Domínguez E, Barriga A, Martin JL. Efficacy of acupuncture for the treatment of fibromyalgia: systematic review and meta-analysis of randomized trials. Open Rheumatol J. 2009;16(3):25–9.

    Article  Google Scholar 

  26. Cao H, Li X, Han M, Liu J. Acupoint stimulation for fibromyalgia: a systematic review of randomised controlled trials. Evid Based Complement Altern Med. 2013;2013:ID 362831.

    Article  Google Scholar 

  27. Yang B, Yi G, Hong W, Bo C, Zhankui W, Yangyang L, et al. Efficacy of acupuncture on fibromyalgia syndrome: a meta-analysis. J Tradit Chin Med. 2014;34(4):381–91.

    Article  PubMed  Google Scholar 

  28. de Souza Nascimento S, Desantana JM, Nampo FK, et al. Efficacy and safety of medicinal plants or related natural products for fibromyalgia: a systematic review. Evid Based Complement Altern Med. 2013;2013:10.

    Article  Google Scholar 

  29. Perry R, Hunt K, Ernst E. Nutritional supplements and other complementary medicines for infantile colic: a systematic review. Pediatrics. 2011;127:720–33.

    Article  PubMed  Google Scholar 

  30. Bruyas-Bertholo V, Lachaux A, Dubois J-P, Fourneret P, Letrilliart L. Quels traitements pour les coliques du nourrisson. Presse Med. 2012;41:e404–10.

    Article  Google Scholar 

  31. Harb T, Matsuyama M, David M, Hill RJ. Infant colic—what works: a systematic review of interventions for breast-fed infants. J Pediatr Gastroenterol Nutr. 2016;62(5):668–86.

    Article  PubMed  Google Scholar 

  32. Gutiérrez-Castrellón P, Indrio F, Bolio-Galvis A, et al. Efficacy of Lactobacillus reuteri DSM 17938 for infantile colic. Systematic review with network meta-analysis. Medicine. 2017;96(51):e9375.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Dobson D, Lucassen PLBJ, Miller JJ, Vlieger AM, Prescott P, Lewith G. Manipulative therapies for infantile colic. Cochrane Database Syst Rev. 2012;(Issue 12):CD004796. https://doi.org/10.1002/14651858.CD004796.pub2.

  34. Gleberzon BJ, Arts J, Mei A, McManus EL. The use of spinal manipulative therapy for pediatric health conditions: a systematic review of the literature. J Can Chiropr Assoc. 2012;56(2):128–41.

    PubMed  PubMed Central  Google Scholar 

  35. Carnes D, Plunkett A, Ellwood J, et al. Manual therapy for unsettled, distressed and excessively crying infants: a systematic review and meta-analyses. BMJ Open. 2018;8:e019040.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Skjeie H, Skonnord T, Brekke M, Klovning A, Fetveit A, Landgren K, et al. Acupuncture treatments for infantile colic: a systematic review and individual patient data meta-analysis of blinding test validated randomised controlled trials. Scand J Prim Health Care. 2018;36(1):56–69.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Anheyer D, Frawley J, Koch AK, Lauche R, Langhorst J, Dobos G, et al. Herbal medicines for gastrointestinal disorders in children and adolescents: a systematic review. Pediatrics. 2017;139(6):e20170062.

    Article  PubMed  Google Scholar 

  38. Sung V, Collett S, de Gooyer T, et al. Probiotics to prevent or treat excessive infant crying. JAMA Pediatr. 2013;167(12):1150–7.

    Article  PubMed  Google Scholar 

  39. Anabrees J, Indrio F, Paes B, AlFaleh K. Probiotics for infantile colic: a systematic review. BMC Pediatr. 2013;13:186.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Urbanska M, Szajewska H. The efficacy of Lactobacillus reuteri DSM 17938 in infants and children: a review of the current evidence. Eur J Pediatr. 2014;173:1327–37.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Xu M, Wang J, Wang N, Sun F, Wang L, Liu XH. The efficacy and safety of the probiotic bacterium Lactobacillus reuteri DSM 17938 for infantile colic: a meta-analysis of randomized controlled trials. PLoS One. 2015;10(10):e0141445.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  42. Schreck Bird A, Gregory PJ, Jalloh MA, Risoldi Cochrane Z, Hein DJ. Probiotics for the treatment of infantile colic: a systematic review. J Pharm Pract. 2017;30(3):366–74.

    Article  PubMed  Google Scholar 

  43. Dryl R, Szajewska H. Probiotics for management of infantile colic: a systematic review of randomized controlled trials. Arch Med Sci. 2018;14(5):1137–43.

    Article  PubMed  CAS  Google Scholar 

  44. Sung V, D’Amico F, Cabana MD, Chau K, Koren G, Savino F, et al. Lactobacillus reuteri to Treat Infant Colic: A Meta-analysis. Pediatrics. 2017;141(1):e20171811.

    Article  Google Scholar 

  45. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76(5):378.

    Article  Google Scholar 

Download references

Acknowledgements

We thank Professor Rachel Churchill for the initial idea to compare the two assessment tools and Dr Chris Penfold for checking domain 4 in the ROBIS tool.

Funding

The work of RP and AW was funded by National Institutes of Health Research (NIHR) Biomedical Research Centre at the University Hospitals Bristol and Weston NHS Foundation Trust. Philippa Davies’ time was supported by the National Institute for Health Research Applied Research Collaboration West (NIHR ARC West) at University Hospitals Bristol NHS Foundation Trust.

The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

Author information

Authors and Affiliations

  1. National Institute for Health Research Bristol Biomedical Research Centre, University Hospitals Bristol and Weston NHS Foundation Trust and University of Bristol, Bristol, UK

    R. Perry & A. Whitmarsh

  2. The National Institute for Health Research Applied Research Collaboration West (NIHR ARC West) at University Hospitals Bristol NHS Foundation Trust, Bristol, UK

    V. Leach & P. Davies

  3. Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK

    P. Davies

Authors
  1. R. Perry
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Whitmarsh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. V. Leach
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. P. Davies
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

RP assessed all papers using AMSTAR-2 and ROBIS and contributed to the paper. PD assessed all papers using AMSTAR-2 and ROBIS and contributed to the paper. VL assessed all papers using AMSTAR-2 and ROBIS and contributed to the paper. AW conducted the inter-rater reliability analyses and contributed to the paper. The authors read and approved the final manuscript.

Corresponding author

Correspondence to R. Perry.

Ethics declarations

Ethics approval and consent to participate

Not relevant.

Competing interests

Rachel Perry was an author on two of the papers under review. Philippa Davies was involved in the development of ROBIS. The other authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Results of AMSTAR-2 for CAM for fibromyalgia reviews

The inter-rater agreement between the three raters is shown in Table 7.

Table 7 Results of AMSTAR-2 for CAM for fibromyalgia reviews
Full size table

Results of ROBIS: CAM for fibromyalgia

The summary of results of ROBIS for fibromyalgia can be seen in Table 8.

Table 8 Inter-rater agreement
Full size table

Inter-rater agreement for fibromyalgia

For AMSTAR-2, 10 out of 16 (62.5%) questions had substantial agreement or higher between reviewers. There was perfect agreement for questions 2 (did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?), 7 (did the review authors provide a list of excluded studies and justify the exclusions?) and 10 (did the review authors report on the sources of funding for the studies included in the review?). The median agreement for all questions was 0.65. Ratings from a reviewer were missing in 20 instances overall.

Ten out of 24 (41.7%) questions for ROBIS had at least substantial agreement. Questions 3.1 (were efforts made to minimise error in data collection?) and 3.5 (were efforts made to minimise error in risk of bias assessment?) had almost perfect agreement. The median agreement for all questions was 0.55. The agreement was different for each ROBIS domain with substantial being the highest agreement (for missing in 6 instances). The raters gave a rating of “no information” in 93 cases. In most of these cases (65), the other two raters gave a different rating. There were 5 instances where all reviewers reported “no information”. The most common questions for “no information” were questions 1.1 (did the review adhere to pre-defined objectives and eligibility criteria? 13 times), 4.2 (were all pre-defined analyses reported or departures explained? 13 times) and 4.5 (were the findings robust, e.g. as demonstrated through funnel plot or sensitivity analyses? 11 times) .

Tables 9 and 10

Table 9 The risk of bias and study quality for each fibromyalgia review
Full size table
Table 10 To compare the distribution of risk of bias and study quality for the fibromyalgia reviews
Full size table

Results of AMSTAR-2: CAM for colic

The inter-rater agreement between the three raters is shown in Table 11.

Table 11 Inter-rater agreement
Full size table

Results of ROBIS: CAM for colic

The inter-rater agreement between the three raters is shown in Table 12

Table 12 Inter-rater agreement
Full size table

Inter-rater agreement for colic

Eight of 16 (50%) AMSTAR-2 questions had substantial agreement or higher. There was almost perfect agreement for questions 2 (did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?), 7 (did the review authors provide a list of excluded studies and justify the exclusions?) and 9 (did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?). The median score for all questions was 0.60. Ratings from a reviewer were missing in 15 instances overall.

Thirteen of 24 (54.2%) ROBIS questions had substantial agreement or higher. There was almost perfect agreement for questions 3.1 (were efforts made to minimise error in data collection?) and 3.4 (was risk of bias (or methodological quality) formally assessed using appropriate criteria?). The median score for all questions was 0.63. The agreement was different for each ROBIS domain with substantial being the highest agreement (for domain 3). The agreement for the risk of bias was moderate. Ratings from a reviewer were missing in 3 instances. There were 66 ratings of “no information”. There were 3 instances where the reviewers were in agreement. In 42 cases, only one reviewer said “no information”. The most common questions were questions 1.1 (did the review adhere to pre-defined objectives and eligibility criteria? 10 times), 3.5 (were efforts made to minimise error in risk of bias assessment? 8 times) and 4.2 (were all pre-defined analyses reported or departures explained? 9 times).

Tables 13, 14 and 15

Table 13 The risk of bias and study quality for each colic review
Full size table
Table 14 To compare the distribution of risk of bias and study quality for the fibromyalgia reviews
Full size table
Table 15 Criteria for assessing confidence in AMSTAR-2 (Shea et al. [20])
Full size table

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Perry, R., Whitmarsh, A., Leach, V. et al. A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2. Syst Rev 10, 273 (2021). https://doi.org/10.1186/s13643-021-01819-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13643-021-01819-x

Keywords

Systematic Reviews

ISSN: 2046-4053