A scoping review of network meta-analyses assessing the efficacy and safety of complementary and alternative medicine interventions

Background Network meta-analysis (NMA) has rapidly grown in use during the past decade for the comparison of healthcare interventions. While its general use in the comparison of conventional medicines has been studied previously, to our awareness, its use to assess complementary and alternative medicines (CAM) has not been studied. A scoping review of the literature was performed to identify systematic reviews incorporating NMAs involving one or more CAM interventions. Methods An information specialist executed a multi-database search (e.g., MEDLINE, Embase, Cochrane), and two reviewers performed study selection and data collection. Information on publication characteristics, diseases studied, interventions compared, reporting transparency, outcomes assessed, and other parameters were extracted from each review. Results A total of 89 SR/NMAs were included. The largest number of NMAs was conducted in China (39.3%), followed by the United Kingdom (12.4%) and the United States (9.0%). Reviews were published between 2010 and 2018, with the majority published between 2015 and 2018. More than 90 different CAM therapies appeared at least once, and the median number per NMA was 2 (IQR 1–4); 20.2% of reviews consisted of only CAM therapies. Dietary supplements (51.1%) and vitamins and minerals (42.2%) were the most commonly studied therapies, followed by electrical stimulation (31.1%), herbal medicines (24.4%), and acupuncture and related treatments (22.2%). A diverse set of conditions was identified, the most common being various forms of cancer (11.1%), osteoarthritis of the hip/knee (7.8%), and depression (5.9%). Most reviews adequately addressed a majority of the PRISMA NMA extension items; however, there were limitations in indication of an existing review protocol, exploration of network geometry, and exploration of risk of bias across studies, such as publication bias. Conclusion The use of NMA to assess the effectiveness of CAM interventions is growing rapidly. Efforts to identify priority topics for future CAM-related NMAs and to enhance methods for CAM comparisons with conventional medicine are needed. Systematic review registration https://ruor.uottawa.ca/handle/10393/35658


Background
The use of complementary and alternative medicine (CAM) interventions is common [1][2][3][4][5][6], and the number of randomized controlled trials (RCT) and systematic reviews related to CAM interventions have previously been shown to be on the rise [7,8]. As physicians are sometimes hesitant to discuss the use of CAM therapies with patients due to a lack of comfort in addressing related questions, there is a need to ensure rigorous scientific evidence of their benefits and harms is available [9,10]. Past research has suggested that reviews of CAM interventions have been associated with certain areas of strength and weakness in terms of rigor relative to systematic reviews of other types of interventions [11], and challenges regarding clinical trial design and priority setting have also been identified [12].
Network meta-analysis (NMA) is a generalization of traditional pairwise meta-analysis [13,14] and the use of NMA has grown rapidly in recent years [15][16][17]. NMA is of considerable value to researchers, analysts, and decision-makers when dealing with clinical scenarios requiring the comparison of multiple alternative therapies, as well as scenarios where there exists both direct and indirect evidence of relevance to the research question at hand [18,19]. Methodologic research related to the conduct of NMA has also grown rapidly, and its use is also supported by helpful implementation tools including reporting guidance, overviews of adapted procedures for judging the strength of evidence, and published considerations for critical appraisal by decision-makers [20][21][22][23].
While the use of NMA for the comparison of pharmacologic interventions is common in the literature [16], the frequency of and approaches to its use for the evaluation of benefits and harms of complementary and alternative medicine (CAM) interventions-whether against each other or relative to other non-CAM interventions-to our awareness, has not been studied. In order to inform comparisons between traditional and complementary therapies, NMA represents a potentially valuable tool to establish relative benefits and harms. In the current study, we used a scoping review approach to establish the extent of published NMAs involving CAM interventions in the literature, to assess their objectives as well as clinical and methodologic characteristics, and to judge the current level of reporting transparency based upon criteria of the PRISMA Extension Statement for Network Meta-Analysis [23]. This information will be of value to establish what topics have been assessed in existing NMAs in the literature, thereby helping to prioritize both research topics as well as methodologic approaches for NMAs involving CAM interventions moving forward for interested physicians, decision-makers, and patients alike. Findings from this review will be informative for researchers and stakeholders seeking to prioritize future topics for CAM-related NMAs and may also allow for the identification of conditions wherein future randomized controlled trials of CAM interventions may be informative in the derivation of comparisons with traditional medical interventions.

Review methods
A protocol for this review was drafted a priori by members of the authorship team. The protocol is available from the University of Ottawa Library's Online Repository (available from https://www.ruor.uottawa.ca/handle/10393/35658). This review has been prepared in consideration of the guidance provided by the PRISMA extension statement for scoping reviews as well as the Joanna Briggs Institute [24,25].

Literature review, eligibility criteria and study selection
Published NMAs involving CAM interventions were identified for the current review using a combination of two approaches. First, three co-authors (AV, PR, ACT) maintained a database of all published NMAs published between 1999 and 2015 based upon a multi-database search strategy (including Medline, Embase, and the Cochrane Database) updated on a quarterly basis, with screening of citations and full texts performed by two independent reviewers; details of the search strategy used to establish the database are provided in Additional file 1. An update of the search was performed on May 29, 2018, with analogous techniques for screening of titles/abstracts and fulltext articles used to identify and include relevant review articles. From the perspective of identifying reviews including NMA, studies selected for inclusion in the database were required to: (a) have used a valid comparison method (such as adjusted indirect comparison, Bayesian model, meta-regression, multivariate meta-analysis, graph theoretical approach); (b) included a minimum of 4 interventions in the network of evidence studied; (c) included a greater number of studies than there were nodes in the network; and (d) included data from RCTs only. For the purposes of the current review, studies identified from the above screening procedures were also reviewed in additional detail in terms of their included interventions to identify reviews that involved one or more CAM interventions; a listing of CAM therapies used during screening is provided in Additional file 2. Screening for reviews incorporating CAM interventions was performed by two independent reviewers (MP, SW). Articles which were focused upon statistical methods investigations relative to NMA were not included in the current review. The process of study selection was summarized using a flow diagram. Only English language reviews were included.

Data collection procedures
A detailed list of information was gathered from each included study that met the study objectives. This information included publication characteristics (i.e., authorship list, year, and journal of publication), core features of each review (e.g., aspects of the research question addressed including study population and endpoints assessed, CAM therapies evaluated), characteristics of each review's network geometry (including whether only CAM interventions were compared in isolation, or whether CAM and non-CAM interventions were established as comparators; and underlying numbers of studies and patients informing analysis); and statistical aspects of analyses performed (including choice of framework [Bayesian vs frequentist], assessment of the consistency assumption, and reporting of secondary measures of treatment effect). The completeness of reporting for each SR/NMA was assessed using the checklist from the PRISMA Extension Statement for NMA [23]. This checklist addresses the 27 core items included in the PRISMA Statement [26] and also addresses 5 additional items specific to the reporting of network metaanalyses (including methods and reporting of findings for each of network geometry inspection and assessment of the appropriateness of the consistency assumption, as well as presentation of a network diagram of the available evidence).

Charting the data
A descriptive approach to summarize the core study characteristics was prepared, along with structured tables and figures to identify salient points of differences noted across studies. A heat map was generated to present the geographic distribution of published reviews (based upon affiliation of each study's lead author), while a word cloud was prepared to assess the relative frequencies with which different CAM interventions were studied in the set of included NMAs. Trends over time in the number of NMAs published with regard to different clinical conditions were reviewed. Bar graphs were generated to evaluate the proportions of included studies adequately addressing individual items of the PRISMA NMA Checklist related to abstract and introduction, methods, results, discussion, and funding status, respectively. Changes in the completeness of reporting were also assessed by year of publication to establish whether the proportions of studies assessed to be of adequate reporting transparency or review methods were improving over time.

Identified literature and general characteristics
In total, literature searching for this review identified a total of 3948 unique abstracts, 90 of which were retained as eligible network meta-analyses that included one or more CAM interventions according to the criteria described earlier . Figure 1 presents a summary of the study selection process. Table 1 presents a detailed summary of the core characteristics of the included reviews, including patients' indication, numbers of studies (and patients) analyzed, endpoints evaluated, key methods used, and review funding.
Year of publication amongst the included reviews ranged from 2010 to 2018 (median 2017; Fig. 2). A total of 35 (39%) were conducted in China, 11 (12.4%) were conducted in the United Kingdom, 8 (9.0%) were conducted in the United States, and 6 (6.7%) were conducted in Germany; 4 (4.5% per country) were conducted in each of Canada, Switzerland, and Thailand, 3 (3.4%) were conducted in each of the Netherlands and Hong Kong, 2 were conducted in each of Italy (2.3%), Malaysia (2.3%), and Brazil (2.3%), and single reviews were conducted in Korea, Sweden, and Greece (see Table 1); Fig. 3 presents a heat map summarizing the distribution of nations producing the set of included NMAs. Funding was public for 57 reviews (64.0%), private/industry sponsored for 3 reviews (3.3%), and no funding was available for support for 12 reviews (13.5%) (see Table 1); funding was unreported for 17 reviews (19.1%). Table 2 provides a listing of the patient indications that were studied within the included reviews, as well as data regarding both the totality of reviews per indication and the evolution of reviews with CAM interventions between 2010 and 2018. A total of 10 were from the realm of mental health, addressing topics such as depressive disorder, post-traumatic stress disorder, postnatal depression, treatment-resistant depression, obsessivecompulsive disorder, psychotic disorders, panic disorder, and attention deficit hyperactivity disorder [34,35,39,45,60,75,76,80,93,108]. A total of 11 reviews related to cancer were identified, including NMAs of interventions for gastrointestinal cancer, pancreatic cancer, acute promyelocytic leukemia non-small cell lung cancer, neurotoxicity from chemotherapy, and cancer-related fatigue [37,54,66,74,82,89,95,105,107,112,113]. Osteoarthritis (including prostatitis) was the subject of 10 reviews [28-30, 40, 42, 73, 78, 101, 103, 114], gastrointestinal infections/disorders were the subject of 6 reviews [61,64,65,67,68,115], cardiovascular disease in 4 reviews [46,69,99,102], topics related to pregnancy, childbirth and newborn health in 4 reviews [47,62,71,83], and a variety of other clinical indications were assessed in 3 or fewer reviews. The number of NMAs overall increased notably from earlier to later years.

Patient indications and outcomes studied
The outcomes studied within each review were also collected. While a narrative overview of these endpoints is not provided here given the extensive nature of this information, a detailed listing for each review has been included in Table 1.

Statistical methods and completeness of reporting
Amongst the included reviews, 60 (66.7%) performed analyses using a Bayesian model for NMA while the remaining 29 (32.2%) used a frequentist approach (see Table 1). Consideration of the appropriateness of the consistency assumption was discussed in 70 (78.7%) reports. In addition to reporting of primary findings using    Table 1). With regard to the completeness of reporting, the proportion of included NMAs adequately addressing each of the 32 items from the PRISMA NMA Checklist is   Table 2 Time horizon of NMA publications by clinical indication   Condition studied  Distribution of reports by year of publication   2010  2011  2012  2013  2014  2015  2016  2017  2018  Total   Acute promyelocytic leukemia  1  1 Attention deficit hyperactivity disorder 1 1 Alzheimer's disease 1 1 2 Antibiotic-associated diarrhea 1 1 Atrophic vaginitis 1 1 Blood pressure reduction 1 1 Bulemia nervosa 1 1 Cancer-related fatigue 1 1 Cerebral infarction 1 1 Chronic ankle instability 1 1 Chronic fatigue syndrome 1 1 Chronic kidney disease 1 1 Chronic kidney disease mineral and bone disorder 1 1 Chronic low back pain 1 1 Chronic obstructive pulmonary disease 1 1 Chronic oro-facial pain 1 1 Chronic pain 1 1 Chronic prostatitis and chronic pelvic pain syndrome 1 1 1 3 Non-alcoholic fatty liver disease 1 1 summarized in Fig. 5 (an overview of the PRISMA NMA Checklist is provided in Additional file 3, while Additional file 4 contains a detailed account of the study-specific assessments). For twenty checklist items (but only one of the 5 added checklist items specific to NMA), reporting was judged to be adequate for 80% or more of the reviews assessed; this included core elements of the abstract, introduction, and methods (specification of eligibility criteria, search information sources, process for study selection, methods for data collection, variables extracted, risk of bias appraisal methods, principal summary measures, methods for meta-analysis), as well as certain components of the findings and discussion sections (numeric details of study selection, provision of a network graph, presentation of study characteristics, presentation of risk of bias data, summary data related to included studies, appraisal of the risk of bias across studies, a summative overview of findings, discussion of study limitations and interpretations). Several other checklist items were associated with less common completeness of reporting. Amongst the 89 included reviews, only 44 (49.4%) identified the report as a systematic review incorporating a NMA (Checklist Item 1). Few studies adequately reported whether a review protocol existed, and where to access the protocol (Checklist Item 5; ; regarding the latter two elements, reporting was also less complete within the results of the included reviews (Checklist Items 22 and 23). Funding and funder roles were also inconsistently reported (Checklist Item 27). With regard to Checklist Items S1-S5 that are specific to NMA, only one exceeded 80% adequate reporting across the included reviews (Checklist Item S3provision of a network graph). Methods used to explore network geometry (Checklist Item S1), methods to assess for inconsistency of direct and indirect evidence (Checklist Item S2), description of the traits of the evidence network (Checklist Item S4) and findings from analyses checking for inconsistency (Checklist Item S5) were adequately reported in totals of 34.8%, 73.0%, 56.8%, and 69.3%, respectively. In reviewing the distribution of the median (IQR) total number of PRISMA items reported over time, findings suggest that the reporting transparency of network meta-analyses has improved slightly over time in NMAs with CAM interventions. In 22 included reviews published prior to 2016 (date chosen in relation to the publication of the PRISMA extension statement for NMA in June 2015), the median (interquartile range) number of items addressed out of 32 (i.e., 27 core items and 5 NMA-related items) was 25 (IQR 23-27.5). In the set of 67 reviews published since the start of 2016, the corresponding median was 26 (IQR 24-28). Totals of 41 (61.2%) reviews published in 2016 and afterward adequately addressed 25 or more checklist items, while the corresponding total amongst those published in 2015 and earlier was 7 (31.8%). With regard to NMA-specific reporting items (S1-S5), improvements were noted in the more recent category of publications for S2 regarding inconsistency methods (79.7% versus 56.0%), S3 regarding provision of network diagrams (96.9% versus and 84.0%) and S5 regarding findings from inconsistency evaluations (70.3% versus 64.0%), while the proportions of studies for S1 and S4 regarding assessment of network geometry patterns were similar across time periods.

Discussion
The growth of NMA as an incrementally important knowledge synthesis methodology for the comparison of healthcare interventions is well established [16]. While its value in informing the comparison of multiple pharmacologic therapies, in particular, is well known, the use of NMA in evaluating the benefits of CAM interventions, to our awareness, has not previously been studied. In the current scoping review, we have enhanced the current understanding of its history of use in the CAM realm. Several interesting findings were identified in the context of this scoping review. First, the annual frequency of NMAs incorporating one or more CAM interventions has risen considerably since 2010, aligning with the type of relative growth observed with NMAs in general. While the largest number of reviews included in this study was produced in China, the diverse range of countries represented was geographically diverse, corroborating the use of NMA to be global in nature. The range of CAM interventions studied and the assortment of medical diagnoses in which they were assessed were also diverse, with certain most common approaches to treatment (including dietary supplements, vitamins, minerals, and East Asian herbal medicines) being observed. From a design perspective, the current review suggests that in many cases, CAM interventions were considered either in separation from conventional medicine (compared only with other CAM therapies) or only a very Graded completeness and transparency of reporting (PRISMA-NMA; n = 89 reviews). A bar chart presenting the judged compliance of included reviews with recommendations from the PRISMA extension statement for NMA is shown, stratified by component of the guidance. The supplemental items (S1-S5) specific to NMA are bolded and underlined on the horizontal axis limited amount of CAM therapies were included in comparisons with conventional medicine. The rationale for both occurrences may potentially be driven by the uncertainty of many researchers as to the benefits that CAM interventions as a whole may potentially offer patients; other plausible rationale may include the stages of disease assessed in reviews (CAM therapies may be tried earlier or later in different cases), the types of benefits that are sought by physicians and patients (e.g., symptom relief versus the impact on disease progression), or concerns regarding potential differences in patient populations (i.e., the potential for systematic differences between those agreeing to receive CAM versus non-CAM interventions). Strategies to enhance their inclusion may, therefore, require greater collaboration amongst CAM experts and producers of systematic reviews to establish more diverse research teams, in particular at the design phase of systematic reviews, to grow the list of comparators for consideration; however, this may not address all existing challenges.
The collection of systematic reviews incorporating NMAs identified in this scoping review offers opportunities in several directions. From the perspective of planning future research, the listing of included reviews may allow organizations with a focus in CAM interventions to (a) identify clinical diagnoses considered highly amenable to CAM therapies where no prior NMA has been conducted, allowing for plans to address a current knowledge gap; (b) identify reviews for high priority indications where a comprehensive comparison amongst CAM therapies has not yet been conducted, with past reviews focused upon only a very limited selection; (c) identify reviews for high priority indications where there remains a clear need to derive treatment comparisons between CAM and conventional medicines; and (d) to consider possible conditions wherein future randomized trials of CAM therapies may be informative. While not discussed in detail in the text of this review, the summary table of past reviews also lists the considered outcomes from past NMAs for consideration by multiple audiences to allow thought as to ways existing information might be helpful or to enhance plans for future research in syntheses related to clinical areas assessed in prior reviews. Surveys indicate that the most commonly used CAM therapies in the US are non-vitamin, non-mineral dietary therapies [117], and this is consistent with the relative prominence of dietary supplements observed in this scoping review. The next most commonly used CAM therapies are deep breathing exercises, yoga, chiropractic or osteopathic manipulation and meditation, and more recent US research also indicates that the percentage of persons using yoga, meditation or chiropractic therapies is increasing [117]. These therapies appeared less often in NMAs, and with increased use, these therapies may be a focus of future research comparisons. CAM therapies are used by a large proportion of people diagnosed with chronic conditions [118], particularly musculoskeletal pain conditions such as arthritis [119]. Although many people who use CAM do so for musculoskeletal pain or mental health [120], many people with musculoskeletal pain conditions who use CAM do not use the CAM to treat pain [121]. Likewise, some of the most commonly used CAM modalities such as dietary supplements or yoga are most frequently used for "wellness" reasons rather than treatment of a condition [122]. Identifying where appropriate CAM therapies could be incorporated into NMAs, therefore, cannot rely only upon the prevalence of use, but rather will also consult with researchers and clinicians to identify gaps in the NMA literature. This scoping review may assist in this identification.
In reviewing the completeness and transparency of reporting of the set of included NMAs, several weaknesses were identified relative to both core elements of PRISMA as well as certain elements specific to the PRISMA Extension statement for NMA; this aligns with past evaluations of published NMAs [123], and efforts to enhance both elements are needed. From a methodologic perspective, further research considering specific elements that relate to the conduct and assumptions underlying NMA may also be relevant. For example, the appropriateness of "lumping" control groups (such as different forms of sham therapy, placebo, and waitlist controls) requires consideration and has been shown previously to potentially introduce bias into the findings of NMAs based upon differential event rates or mean values between sources of control [124][125][126][127]. Furthermore, careful consideration as to whether the study populations enrolled in trials of CAM interventions may differ in important ways relative to those enrolled in trials of conventional methods may also present challenges to the transitivity assumption. In our analyses that looked at trends in reporting completeness based upon PRISMA NMA over time, the median (and IQR) numbers of elements addressed were similar before and after 2016, though the proportions of studies before and after this date that addressed totals of > 25 items (61.2% versus 31.8%) and > 30 items (4.5% versus 0%) both were improved in the latter group.
There are certain limitations to this review to be noted. First, while this scoping review set out to map the conditions studied, CAM interventions evaluated, reporting completeness and other elements, judgements as to the appropriateness of methods for NMA and the completeness of interventions compared in NMAs (from a clinical relevance perspective) were not drawn; while of interest, these were considered to be beyond the goals for this research. Second, while certain characteristics of the NMAs were associated with failures to provide related information within the article text, we did not contact authors for these details, instead, we rely upon what was described only in the article. Last, we did not search registration records for ongoing systematic reviews that may be oriented toward the comparison of CAM therapies or involve comparisons between CAM and conventional medical interventions, and thus the data presented here may underestimate the extent of ongoing NMA evaluations involving CAM therapies.

Conclusion
The application of NMA methods to inform comparisons of CAM interventions has grown rapidly in recent years, and the diversity of interventions assessed and conditions studied is diverse. Given the prevalence of use of CAM interventions, particularly for musculoskeletal conditions and mental health, future efforts to incorporate comparisons in NMAs with conventional medicines and to identify and address the methodologic challenges of NMA in this setting are worthwhile for the comprehensive identification and comparison of treatment options. This review may serve as a starting point from which future research initiatives related to the evaluation of CAM interventions can be prioritized.
A completed PRISMA for Scoping Reviews Checklist is provided in Additional file 5 to document the completeness of reporting of this review.