Table 6 Summary of the ROBIS domains

Perry (2010)

Low: There was no mention of a review protocol but did mention that the inclusion/exclusion criteria were pre-defined. Some additional searching took place; reference lists and other reviews were hand-searched.

Low: Although the search included appropriate databases to identify published studies, searches did not included trial registries or conference reports. The review was restricted to published studies. Two reviewers looked at full texts, but this was not specifically stated for abstract screening.

Low: Two reviewers independently performed data extraction and risk of bias assessment. Risk of bias was assessed using appropriate criteria (Jadad score [39]) with allocation concealment being assessed in addition.

Some study characteristics were extracted (main table), but information was missing on participants. Appropriate results appear to have been collected although this is not completely clear.

Unclear: There was heterogeneity; thus, no meta-analysis was performed. Each study was discussed and evaluated in detail, and a sufficient synthesis occurred. The results of the risk of bias assessment were reported in full. This narrative review assesses the results appropriately and the conclusion reflects this.

Low: The main concerns arising from this were the potential for missed studies through not includ unpublished papers. The conclusions seem fair in relation to these considerations.

Boehm (2014)

High: There was no mention of a review protocol or pre-specification of review objective. There were some concerns regarding the specification of the eligibility criteria with regard to diagnosis of fibromyalgia. No specific list of outcomes stated.

Low: Although the search included appropriate databases to identify published studies, searches did not included trial registries or conference reports. Limited details on the search strategy. The term ‘homeopathy’ was used which would not pick up ‘homeopathic’.

Low: Two reviewers independently performed data extraction and risk of bias assessment. Risk of bias was assessed using appropriate criteria (Cochrane risk of bias [42]). Appropriate study characteristics were extracted, and appropriate results appear to have been collected.

High: One Fisher study (1986) was not included in the synthesis, but it unclear why it was excluded. Combining RCTs with non-RCTs will introduce bias.

High: The discussion is mostly cautious although the final sentence is a bit over-confident.

Some attention given to inclusion of different study designs and the ambiguous definition of homeopathic remedy.

Mayhew (2007)

Low: There was no mention of a review protocol or pre-specification of review objective. There was some concern regarding the specification of the eligibility criteria with regard to outcomes as no outcomes were mentioned.

High: Although the search included appropriate databases to identify published studies, searches did not included trial registries or conference reports. Limited details were available for the search strategy; the full search was not reported. Methods used to screen references and select studies for inclusion were not reported.

High: Two reviewers independently performed data extraction. It was unclear if the two assessed risk of bias. Risk of bias was assessed using appropriate criteria (Jadad score [39]) although allocation concealment is not assessed. There was some reporting of means and percentage differences between groups but not for every study. They also failed to define outcome at the start.

Low: There was limited result information given and as there was no protocol; we cannot check outcomes that were intended to be assessed. This is not really a synthesis, more like a list of finings.

Low: Although some of the domains had issues, the conclusion does take into account some of the weaknesses of the studies and does not overemphasise any positive findings.

Daya (2007)

High: There was no mention of a review protocol or pre-specification of review objective. Lack of detail on eligibility criteria and limited to English language.

High: Although the search included appropriate databases to identify published studies, searches did not include trial registries or conference reports. Limited details were available for the search strategy (no mention of MeSH headings). It appears that the review was restricted to published studies. Methods used to screen references and select studies for inclusion were not clearly reported and appeared to be done by just the author, so no cross-checking.

High: One reviewer performed data extraction and risk of bias assessment. Risk of bias was assessed using appropriate criteria (Stricta [104]). Appropriate study characteristics were extracted (main table) but only P values appear to have been extracted.

High: The results of the individual studies are reported without any real attempt at a synthesis. The quality scale also includes other items which is likely to affect the overall score. Conflicting results between the highest quality studies suggests the findings were not robust.

High: The conclusions seem appropriate for the limitations of the evidence. Main concerns are the potential for missing studies from the limited search and mainly due to a single person conducting the review with no cross-checking.

Langhorst (2010)

Low: There was no mention of a review protocol or pre-specification of review objective. However, there were very detailed eligibility criteria. The search was restricted to fully published studies. The type of acupuncture was restricted to ‘verum’ acupuncture (inserting needles). Acupressure, TENS, and infrared light were excluded, which are appropriate exclusions.

Low: Although the search included appropriate databases to identify published studies, searches did not included trial registries or conference reports. Reference lists, other systematic reviews, and evidence-based guidelines were also searched. The search looks reasonable and is transparent. Methods used to screen references and select studies for inclusion were clearly reported.

Low: Two reviewers extracted data, but it does not state directly in the text if two reviewers independently performed risk of bias assessment (van Tulder score [58]). Risk of bias was assessed using appropriate criteria. Appropriate study characteristics were extracted (main table).

Low: There is a slight error in reporting of results in text and in forest plots. Publication bias could not be assessed due to low number of studies. Sensitivity analysis looked at those with low risk of bias did not show an effect in the meta-analysis.

Low: Main concerns arising from this review were the potential for publication bias though only including published studies. It did not clearly state whether two people assessed risk of bias. However, the analysis and sensitivity analysis were appropriate and thorough and helped the authors draw more conservative and appropriate conclusion.

Martin-Sanchez (2009)

Low: There was no mention of a review protocol or pre-specification of review objective. Inclusion criteria were brief but there did not appear to have any restrictions.

High: Although the search included appropriate databases to identify published studies, searches did not include trial registries or conference reports. Limited details were available for the search strategy. No MeSH terms were mentioned, and full search was not reported. They did not search any CAM databases. Limited number of references identified. There was no information on restrictions e.g. date, publication format, language. Methods used to screen references and select studies for inclusion were not clearly reported.

High: There was insufficient reporting on all aspects of data collection, risk of bias assessment and results.

High: It was unclear why studies were not included in the meta-analysis. The first meta-analysis consisted of 4 of 6 studies. Heterogeneity was discussed briefly. There was no quality assessment, so no insight into methodological quality or risk of bias. No sensitivity analysis.

High: None of the limitations identified were considered in the discussion. Think it is highly likely that reviewers have missed studies. No consideration of study quality, which is a key component of systematic reviews.

Deare (2013)

Low: Cochrane reviews are required to have a protocol which is peer assessed before the review can commence. No restrictions on language and publication type. There were restrictions in studies that did not provide adequate details of control group. Conference abstracts appear to be excluded (see flow diagram).

Low: There were no major concerns with this section. It appears just one reviewer did the screening of titles and abstracts though.

Low: No concerns with this section. Two reviewers independently performed data extraction and risk of bias assessment. Risk of bias was assessed using appropriate criteria (Cochrane [42]). Appropriate study characteristics were extracted (main table), and appropriate results appear to have been collected.

Low: No major concerns; however, one thing to highlight is concerning robustness of the findings. This judgment depends on the comparison:

Acupuncture V no acupuncture

(just 1 study)

Acupuncture V placebo/sham (robust findings)

Low: The conclusion was appropriate and addressed the concerns raised.

Cao (2013)

Low: There was no mention of a review protocol or pre-specification of review objective.

However, outcomes were not clearly specified and did not appear to constitute an objective pre-specified list.

High: Although the search included appropriate databases to identify published studies, searches did not included trial registries or conference reports.

The search strategy appeared comprehensive, but it was unclear if both MeSH and text word used. It appears that the review was restricted to published studies although this was not completely clear. Methods used to screen references and select studies for inclusion were clearly reported.

Low: Two reviewers independently performed data extraction and risk of bias assessment. Risk of bias was assessed using appropriate criteria (Cochrane ROB [42]). Appropriate study characteristics were extracted (main table), and appropriate results appear to have been collected although this is not completely clear.

Low: Unclear if MA included all suitable papers. Lack of guidance on ROBIS tool about how to appropriately consider robustness of quality on results when there is insufficient numbers of studies.

Low: The conclusion seemed to address all the concerns raised in the other domains.

Yang (2014)

Low: There was no mention of a protocol, but there was detailed pre-specification of review objectives. Over all, there were limited concerns with this domain but the texts were restricted to Chinese and English which should be ok for an acupuncture review.

Low: Although the search included appropriate databases to identify published studies, searches did not include trial registries or conference abstracts. Methods used to screen and select studies for inclusion were clearly reported. The language restriction has been dealt with in domain 1.

High: Risk of bias was assessed using Cochrane criteria; however, both Harris (2005) and Guo (2005) have been assessed twice and have very different risk of bias scores despite being the same study. This is confusing and questions whether errors have been made in the assessment.

High: In the meta-analysis, the reviewers could have synthesised the VAS and NRS in the same forest plot. Again, there is an issue of Harris and Guo appearing twice in the plots even though they are the same study (with the same control arm). Results are not robust as there are insufficient studies to assess robustness.

High: ‘Despite the methodological limitations the superiority of acupuncture in the treatment of FMD cannot be denied’ is an overstatement. The flaws in the assessment of risk of bias and the untrustworthy results from the meta-analysis make this review of high risk of bias.

Ernst (2009)

High: There was no mention of a review protocol but did mention that the inclusion/exclusion criteria were pre-defined. No mention of patients with a formal diagnosis of fibromyalgia.

Unclear: Although the search included appropriate databases to identify published studies, searches did not included trial registries or conference reports. Departmental files were searched (which could be a biased selection) and hand-searching took place. The full search strategy was not provided.

Methods used to screen references and select studies for inclusion were not clearly reported. It was not reported how many reviewers screened titles and abstracts.

High: Two reviewers independently performed data extraction and risk of bias assessment. Risk of bias was assessed using appropriate criteria (Jadad score [39]) although allocation concealment was not assessed.

Some study characteristics were extracted (main table), but information was missing on participants.

Appropriate results (when available) appear to have been collected although this is not completely clear.

Unclear: The results of the risk of bias assessment were reported in full; however, allocation concealment was not assessed. This narrative review assesses the results available; however, no numerical results given. Heterogeneity was not formally assessed. The results from Wise and Walsh were not reported in the primary study; thus, a possible source of bias as their results could affect the overall conclusions.

Unclear: The conclusions are inconclusive which is reasonable based on the evidence available. The possibility of missing studies is discussed. The studies are rated low quality so item on allocation concealment is unlikely to have changed this (Jadad scale).

de Souza Nascimento

(2013)

Low: The review did not refer to a protocol; however, the inclusion/exclusion criteria were pre-defined. The review was restricted to English language papers only. Not much grey literature searching took place.

Low: The search included appropriate databases to identify published studies. Reference lists were hand-searched. The search looks reasonable and is transparent, although CAM-specific databases were not searched.

It is unclear whether unpublished papers would be identified. It appears that the review was restricted to published studies. Trial registries were not searched.

Methods used to screen references and select studies for inclusion were not clearly reported.

Low: It states in the text that two reviewers independently performed risk of bias assessment. Risk of bias was assessed using appropriate criteria (both Jadad and Cochane). Insufficient study characteristics were extracted, and there was not enough information about the actual results obtained—just ‘a significant difference was found.’

No actual data provided, just a summary of the result. Unclear which results were used to come to these conclusions. A pilot study was mentioned (Triaste) but no further information as to why this was excluded.

High: No protocol provided

Heterogeneity not discussed. It was unclear why certain studies could not be combined. Narrative synthesis of results mentioned the direction of effect but no information about the size of the effect.

Low: Main concerns arising from this were the potential for publication bias through only including published studies and restricting to English language only. There was a tendency to emphasise the positive findings. They made no mention of the small number of studies or the risk of bias in those studies when interpreting the results.

Holdcroft (2003)

High: There was no mention of a review protocol and ambiguous eligibility criteria. There were no fibromyalgia criteria or any outcomes listed. One restriction was to only include those studies judged as good quality of reporting.

High: Although the search included appropriate databases to identify published studies; searches did not included trial registries or conference reports. Search strategy was not available and they have not put the term for homeopathy in (although they do retrieve one study on homeopathy). There are odd search dates for Embase and CINAHL which restrict the search. There is no information about study selection.

High: There was no information about participants or how outcomes were measured. Little information about dose and nothing on study design. One reviewer performed data extraction and quality assessment. The CONSORT checklist was used as a quality assessment tool which is inappropriate. No results are displayed in the table or results section just a statement ‘differed significantly’.

Unclear: No numerical synthesis due to heterogeneity; there was no flow diagram and no list of included outcomes, so it is unclear whether results of studies were not included that should have been (possibility of ‘cherry-picking’ the results). Study design and quality is considered in the narrative; however, the CONSORT checklist is an inappropriate scale to assess quality.

High: The conclusion was suitably cautious but no mention of limitations identified in domains 1-3.

Baronowsky (2009)

High: There was no mention of a review protocol or pre-specification of review objectives. No mention of comparators or outcome of interest. Articles were restricted to English and German languages only which may have missed some papers (particularly Chinese). Nutritional, Herbal medicine and hormonal supplements were excluded from the review.

High: Although the search included appropriate databases to identify published papers, and the terms appears to cover all the CAM therapies that were needed (although no MeSH terms listed), it appears this restricted to published papers. Studies are likely to have been missed due to not searching beyond electronic databases. Details of the screening process were not clearly reported.

Unclear: Quite a few items on the quality assessment checklist are not about quality so this will affect the score. Also, it is not clear how many people assessed quality. Limited information reported on participants. Insufficient results are presented, and actual results (means, SDs) were not reported. P values were reported occasionally within a statement mentioning significance.

Low: There was insufficient reporting of outcomes evaluated and the numerical results. This is particularly an issue when there is no meta-analysis available.

Unclear: Overall, the results show a positive trend in favour of acupuncture. Which might be overstating the findings a bit. The possibility of missing studies is discussed however.

De Silva (2010)

High: There was no mention of a review protocol and a limited pre-specification of review objective. Inclusion was restricted to studies a complementary medicine substance in the UK which restricts this review. It was also restricted to English language.

High: Although the search included appropriate databases to identify published studies, limited details were available for the search strategy. The RCT filter was very basic and likely to miss some trials. It appears that the review was restricted to published studies although this was not completely clear. The search was restricted to electronic databases.

Only 60 references were identified which seems quite limited. Methods used to screen references and select studies for inclusion were clearly reported.

High: Not all data was provided, e.g. results of some studies were not reported. Some P values reported in text. No information in the methods section about results data to be collected. One reviewer performed data extraction and risk of bias assessment, and this was checked. Risk of bias was assessed using Jadad only which is a limitation (no allocation concealment).

Unclear: They seemed to have reported the same number of results as number of studies although not all P values given. No pre-defined analysis. No description of outcomes of interest given in the paper so impossible to judge whether the paper should be included or not.

No mention of heterogeneity but no meta-analysis completed so assumed this was an issue. Only a single trial for some CAMs, so no synthesis of any type was possible.

Restricting to CAM administered just in the UK restricts the generalizability of the results.

Low: Rationale for risk: the small number of studies, methodological limitations and limiting the search to English language only. Use of the Jadad scale was another issue. However, the conclusion does say there is insufficient evidence available.

Terhorst (2011, 2012)

Low: There was no mention of a review protocol and a limited pre-specification of review objectives. In CAM research, there are often lots of Chinese papers, so excluding non-English papers is risky.

High: Although the search included appropriate databases to identify published studies, there were limited details available on the search strategy.

There was a broad range of sources of references searched; searches also included dissertations.

Methods used to screen references and select studies for inclusion were reported.

Low: Unclear if two reviewers extracted data although it states that two assessed risk of bias. Risk of bias was assessed using appropriate criteria (Cochrane GRADE [42]) although adaptions were made. For a review of this size, appropriate study characteristics were extracted (main table) and appropriate results appear to have been collected.

High: Meta-analysis did not include a sensitivity analysis based on quality. Heterogeneity was not assessed. Studies excluded from the analysis were explained. They pooled the data but did not report on how they combined the effect sizes. There was no synthesis in categories where there were less than 5 studies.

High: There was a tendency to be over-positive about the results in general considering the limitations of the search and restricting to English language only limits this review.