Skip to main content
Download PDF

Bibliometric and content analysis of the Cochrane Complementary Medicine Field specialized register of controlled trials

Systematic Reviews volume 2, Article number: 51 (2013) Cite this article

Abstract

Background

The identification of eligible controlled trials for systematic reviews of complementary and alternative medicine (CAM) interventions can be difficult. To increase access to these difficult to locate trials, the Cochrane Collaboration Complementary Medicine Field (CAM Field) has established a specialized register of citations of CAM controlled trials. The objective of this study is to describe the sources and characteristics of citations included in the CAM Field specialized register.

Methods

Between 2006 and 2011, regular searches for citations of CAM trials in MEDLINE and the Cochrane Central Register of Controlled Trials (CENTRAL) were supplemented with contributions of controlled trial citations from international collaborators. The specialized register was ‘frozen’ for analysis in 2011, and frequencies were calculated for publication date, language, journal, presence in MEDLINE, type of intervention, and type of medical condition.

Results

The CAM Field specialized register increased in size from under 5,000 controlled trial citations in 2006 to 44,840 citations in 2011. Most citations (60%) were from 2000 or later, and the majority (71%) were reported in English; the next most common language was Chinese (23%). The journals with the greatest number of citations were CAM journals published in Chinese and non-CAM nutrition journals published in English. More than one-third of register citations (36%) were not indexed in MEDLINE. The most common CAM intervention type in the register was non-vitamin, non-mineral dietary supplements (e.g., glucosamine, fish oil) (34%), followed by Chinese herbal medicines (e.g., Astragalus membranaceus, Schisandra chinensis) (27%).

Conclusions

The availability of the CAM Field specialized register presents both opportunities and challenges for CAM systematic reviewers. While the register provides access to thousands of difficult to locate trial citations, many of these trials are of low quality and may overestimate treatment effects. When including these trials in systematic reviews, adequate analysis of their risk of bias is of utmost importance.

Peer Review reports

Background

Complete identification of eligible controlled trials is an essential step in conducting a systematic review, and finding and collecting citations of controlled trials have been aims of the Cochrane Collaboration from its inception 20 years ago. As part of this mission, the Collaboration developed the Cochrane Central Register of Controlled Trials (CENTRAL), a searchable database of citations of controlled trials [1]. The Collaboration has agreements with the publishers of MEDLINE and EMBASE, ensuring that all citations of controlled trials from those databases are republished in CENTRAL. Cochrane entities (e.g., Cochrane Review Groups) also regularly submit citations of controlled trials to CENTRAL, ensuring that CENTRAL contains trial citations from multiple sources, including not only MEDLINE and EMBASE, but also regional and subject-specific databases, as well as trial citations not included in databases. Because of the Collaboration’s extensive efforts at identification of trials from a range of sources, CENTRAL is considered to be the most comprehensive source of citations of controlled trials for inclusion in systematic reviews [1].

The complete identification of eligible controlled trials can be particularly challenging for systematic reviews of CAM interventions. The disadvantage of relying exclusively upon sources such as MEDLINE for trial identification is illustrated by research conducted by Egger and colleagues. Egger and colleagues [2] analyzed the characteristics of 1,635 controlled trials included in a group of n = 159 systematic reviews, including both conventional and CAM-related reviews. They found that in CAM-related systematic reviews, the proportion of non-MEDLINE-indexed trials (41%) was approximately twice that proportion seen in conventional medicine systematic reviews (21%). If the systematic review authors had searched only MEDLINE for trials, they would have missed many trials, possibly including some important trials, and the proportion missing from CAM reviews would have been double the proportion missing from conventional medicine reviews. Earlier research using various ‘gold standard sets’ of known trials for specific CAM interventions found that the percentage of known trials included in MEDLINE was 58% for acupuncture trials [3], 31% for ginkgo trials [4], and 17% for homeopathy trials [4]. Ensuring that CENTRAL contains both MEDLINE and non-MEDLINE citations of CAM-related controlled trials is therefore important for the unbiased conduct of Cochrane systematic reviews of CAM interventions.

The CAM Field maintains a specialized register of citations of controlled trials of CAM interventions, which is a ‘subregister’ of CENTRAL. In 1998, a bibliometric analysis of the CAM Field specialized register described it as containing 3,774 controlled trials [5]. In 2006, the CAM Field began an active program to improve the scope and size of the CAM Field specialized register. CAM Field staff and international partners in this endeavor performed searches of bibliographic databases and paper journals, and CAM Field staff performed extensive quality checks and de-duplication of all identified citations. As a result of these efforts, the CAM Field register contained a total of 43,310 CAM Field specialized register citations as of Issue 1, 2012, of The Cochrane Library. CAM Field specialized register citations can be retrieved from the Cochrane Central Register of Controlled Trials (CENTRAL) database of The Cochrane Library by searching for the tag ‘SR-COMPMED’ in all text. The objective of this study is to describe the sources and characteristics of the trial citations included in the CAM Field specialized register.

Methods

In 2011 we drew up a detailed protocol (see Additional file 1) that described the eligibility criteria for including citations and the methods we had used to build the register. The protocol also pre-specified the methods we planned to use in examining the characteristics of the citations in the register. A summary of these methods, as well as the details of contributions of trial citations by international collaborators, is presented below.

Eligibility criteria

All citations in the CAM Field specialized register are required to meet the following two inclusion criteria: (1) they must be reports of controlled trials, and (2) they must be CAM-related.

We considered controlled trials to be studies meeting the inclusion criteria for CENTRAL that were formulated and agreed upon in November 1992 and are published in Chapter 6.3 of the Cochrane Handbook [6]. We considered trials to be CAM-related if they described interventions that are outside the practices and theories of disease and healing that are intrinsic to the conventional Western medical model [7]. To retrieve citations of CAM trials from MEDLINE and CENTRAL, we relied upon the CAM on PubMed search strategy, which was jointly developed by the US National Library of Medicine and the US National Institutes of Health, National Center for Complementary and Alternative Medicine, and introduced in PubMed in 2001 [8, 9]. In cases where some uses of the intervention are accepted within conventional Western medicine and others are not (e.g., vitamin supplementation), the CAM on PubMed search strategy generally does not distinguish between conventional and unconventional uses of an intervention. Therefore, for classifying interventions as conventional or CAM, we followed the same operational criteria we had previously developed for classifying Cochrane systematic reviews as conventional or CAM [10]. Some of the major decisions about the scope of CAM were as follows: we excluded vitamins and other supplements that are administered parenterally in hospital settings, we excluded dietary supplementation for treatment or prevention of medically diagnosed deficiency states (e.g., iron supplementation for preventing or treating iron deficiency), and we excluded vitamin supplements for preventing or treating disease in countries where vitamin deficiency is widespread (e.g., vitamin A for treating measles in children in Niger). We included vitamins for other conditions, even vitamins that are accepted for the prevention or treatment of specific disorders (e.g., folic acid for preventing neural tube defects). In general, we decided that we should be over-inclusive rather than under-inclusive with vitamin therapies, aside from the three major exclusions detailed above, and therefore some of the vitamin trials in the database would not be accepted as CAM by most people. Finally, we excluded exercise interventions with the exception of mind-body exercise (e.g., yoga), and we excluded conventional psychotherapies. A full description of the CAM Field operational definition of CAM has been published previously (see Additional file 2) [10].

Methods for building the register of trials

We began the expansion of the CAM Field specialized register of trials by building upon the reference management database of nearly 5,000 CAM controlled trial citations developed during the 1990s and early 2000s by Vickers and colleagues [5]. In 2006, we began regular searches of MEDLINE in PubMed using the CAM on PubMed search strategy. In 2008, an information specialist translated the CAM on PubMed search strategy into a format for use in CENTRAL, and we replaced searches of PubMed with regular searches of CENTRAL. The rationale for replacing searches of PubMed with searches of CENTRAL is that CENTRAL includes not only controlled trial citations from MEDLINE, but also controlled trial citations from multiple other sources. These other citations, which may be from other databases or from difficult to locate sources such as trial proceedings, are identified by Cochrane contributors around the world and contributed to CENTRAL. All Cochrane groups then search CENTRAL in order to identify relevant citations that others have contributed. We began by searching CENTRAL from inception and then searched newly added citations in each subsequent issue of CENTRAL.

An important subset of CAM is traditional medicine, defined by the World Health Organization as “the sum total of knowledge, skills and practices based on the theories, beliefs and experiences indigenous to different cultures that are used to maintain health, as well as to prevent, diagnose, improve or treat physical and mental illnesses. Traditional medicine that has been adopted by other populations (outside its indigenous culture) is often termed alternative or complementary medicine” [11]. Different countries often have their own forms of traditional medicine (e.g., traditional Chinese medicine, traditional Korean medicine). Because we did not have access to the traditional medicine trial reports that are published in regional or national databases and journals, in 2008 we began contacting Cochrane colleagues and contributors to solicit the contributions of citations of traditional medicine trials published in their regions. These efforts are described below. Searches of PubMed, and then of CENTRAL, were thus complemented with searches of bibliographic databases and journals conducted by several international groups who contributed citations of trial reports to the CAM Field for inclusion in the CAM Field register. Citations provided by contributing organizations were not restricted by publication year. As described below, two of these contributing groups also provided PDFs of the full text publications for all identified citations. Collaborators are listed below in order of numbers of citations submitted to the CAM Field for the specialized register.

Traditional Chinese medicine (TCM) trials identified by Chinese collaborators

Beginning in 2008, staff at the Center for Evidence-Based Medicine of the Beijing University of Chinese Medicine, under the direction of Jianping Liu, searched both electronic databases and Chinese journals to identify citations of controlled trials of TCM interventions. The journal titles, article titles, and abstracts (if available) of all identified citations were translated into English, entered into a reference management database with added topic keywords, and submitted to the CAM Field for inclusion in the specialized register. The full text report was also submitted to the CAM Field for each citation in a PDF format.

Trials from CAM-specific databases identified by Canadian collaborators

In 2008, information specialists under the direction of David Moher of the Ottawa Hospital Research Institute undertook a project to search several specialized databases for difficult to locate controlled trials of CAM interventions, and these searches were replicated in 2010 [12]. All identified citations were imported into a reference management database, information about the source database and the type of CAM intervention was included for each citation, and the database was submitted to the CAM Field for inclusion in the CAM Field specialized register.

Traditional medicine trials identified by Korean collaborators

In 2010, researchers at the Korea Institute of Oriental Medicine, under the direction of Myeong Soo Lee, searched both electronic databases and journals to identify citations of controlled trials of traditional medicine interventions conducted in Korea and primarily published in non-MEDLINE journals. Initial searches were focused on identifying trials of acupuncture [13] and ginseng, and were then expanded to include all other CAM interventions. The journal titles, article titles, and abstracts (if available) of all identified citations were translated into English and entered into a reference management database with topic keywords, and the citations were submitted to the CAM Field. For each citation, the full text publication was also submitted to the CAM Field in a PDF format.

Kampo trials identified by Japanese collaborators

Kampo is the Japanese adaptation of traditional Chinese medicine. While Kampo uses most of the interventions of Chinese medicine, including acupuncture and moxibustion, its primary focus is on the study and evaluation of traditional herbal medicines. In 2001, the Japan Society for Oriental Medicine undertook a project to collect controlled trial evidence on Kampo interventions through searches of both electronic databases and journals [14]. As randomized controlled trials of Kampo interventions are identified, structured abstracts are prepared for each trial, and the citations and structured abstracts are published online in English. In March 2011, one of the leaders of this initiative, Kiichiro Tsutani of the University of Tokyo, provided permission for the CAM Field staff to incorporate the citations associated with these Kampo trials into the CAM Field specialized register together with links to the online structured abstracts.

Ayurveda and other CAM-related trials identified by Indian collaborators

The South Asian Database of Controlled Clinical Trials (SADCCT) is an online database of citations of controlled trials that have been conducted in countries for which the South Asian Cochrane Network & Centre is the reference Cochrane Centre, including Afghanistan, Bangladesh, Bhutan, India, the Maldives, Nepal, Pakistan, and Sri Lanka. The SADCCT was developed by searching South Asian journals and conference proceedings for all controlled trials [15]. In 2011, staff at the South Asian Cochrane Network & Centre, under the direction of Prathap Tharyan, identified and forwarded citations of South Asian trials of Ayurveda, Unani, Siddha, and other CAM interventions included in the SADCCT, and in 2012 staff at the CAM Field identified additional trial citations from the online SADCCT.

CAM trials identified by African collaborators

The African Trials Register is a database of citations of controlled trials conducted in Africa. It has been developed at the South African Cochrane Centre by the Cochrane Centre staff searching African journals and electronic bibliographic databases [16, 17]. In 2011, citations of CAM-related trials included in the African Trials Register were identified by Elizabeth Pienaar, information specialist at the South African Cochrane Centre, and a database of citations was forwarded to the CAM Field.

Methods for examining the contents of the register of trials

In August 2011, we suspended additions of new search results to the CAM Field specialized register and began an intensive program of cleaning and updating the register in preparation for analysis, focusing on detection and removal of any non-CAM or non-controlled trial citations, deduplication of register citations, identification of whether each register citation was present or absent in MEDLINE, and standardization of journal names (see Additional file 1 for details of procedures used). Because EMBASE is a second major database from which controlled trials are automatically downloaded to CENTRAL, we also wished to characterize the EMBASE coverage of CAM Field register citations. However, register citations do not contain EMBASE identifiers, and we did not have the resources to comprehensively check all citations for presence in EMBASE. We therefore estimated the proportion of register citations present in EMBASE by taking a random sample of 200 register records and searching EMBASE for each citation in the sample. We used the same random sample to estimate the overlap in coverage between MEDLINE and EMBASE, and the proportion of register citations not present in either MEDLINE or EMBASE.

We sorted journal titles by frequency and the 100 journals associated with the greatest number of trial citations were classified as either CAM or conventional in focus, using the classification method described in the protocol (see Additional file 1). For each of the 25 CAM and 25 conventional journals with the greatest number of trial citations, we determined the language of publication and whether the journal was indexed in MEDLINE. In addition to characterizing the journals with the greatest number of citations, we also examined the distribution of citations across all journals in the register to determine to what extent citations are scattered across journals overall.

To characterize register citations by CAM intervention, we chose 21 different types of CAM interventions within five broad categories. We based the 5 broad categories and the 21 intervention types within categories upon the CAM Field topics list for Cochrane reviews of CAM interventions [10] as well as other classifications of CAM interventions (e.g., the classifications of CAM interventions used in the 2007 NHIS survey of use of CAM in the United States [18]). We then developed subject searches for each of the 21 types of CAM interventions by parsing the 2006 translation of the CAM on PubMed search strategy into the CAM intervention topic areas. We also consulted additional sources to identify any supplementary terms and to help understand and delineate between the CAM intervention topic areas. Searches were run and tested in MEDLINE, using each relevant term. An information specialist developed the MEDLINE version of each search strategy, which was then peer reviewed by an independent information specialist using the PRESS standard [19]. Searches were then adapted so that they could be used to search the reference management database containing the register. MEDLINE Medical Subject Heading (MeSH) terms and free text terms were sought in all database fields. The search strategies for identifying each of the 21 different types of CAM interventions are included as an additional file so that these strategies will be publicly available to CAM systematic reviewers developing searches for any of these classes of therapies (see Additional file 3).

To characterize register citations by medical conditions, we used the 25 categories of medical conditions listed in the browse list on the home page of The Cochrane Library[20]. We developed subject searches for each medical condition category by consulting Cochrane reviews grouped under each of these 25 categories to ascertain relevant search terms and concepts, and additional sources to identify any supplementary terms. Searches were run and tested in MEDLINE in the same manner as described above for the CAM intervention searches, although these searches on medical conditions were not peer reviewed. Search strategies were then adapted so that they could be used to search the CAM register and augmented with additional free text synonyms for relevant medical conditions. Both MeSH and free text terms were sought in all reference management database fields.

Results

At the time of our analysis, the CAM Field register of trials included 44,840 citations of CAM trials, which represented approximately 6% of the total number of all trial citations in CENTRAL [21]. Of these 44,840 CAM trial citations, 15,990 (36%) are not included in MEDLINE. Among the random sample of 200 register citations checked for EMBASE status, 63/200 (31.5%; 95% CI 25% to 38%) are not included in EMBASE. Among these 63 citations not included in EMBASE, only 7/63 (11.1%; 95% CI 3% to 19%) are included in MEDLINE. Among the 69/200 sample citations not included in MEDLINE, only 13/69 (18.8%; 95% CI 10% to 28%) are included in EMBASE. Overall, 56/200 (28%; 95% CI 22% to 34%) of sample citations are not included in either MEDLINE or EMBASE.

There is a substantial increase in the numbers of trial citations published for each 5-year time period, and the majority of the citations are from more recent publication years (see Figure 1). This increase in the numbers of trial citations included in the CAM Field register over publication year time periods corresponds to a similar increase over publication year periods seen overall in CENTRAL [22].

Figure 1
figure 1

Number of citations in the CAM Field specialized register by citation publication year. * The drop in the number of trial citations for 2007–2011 may be partially due to incomplete indexing of trials published in this recent time period and/or may be partially due to our having suspended additions of new search results to the CAM Field specialized register in August 2011.

Full size image

The most common languages in the register after English are Chinese (n = 10,376; 23%), German (n = 963; 2%), Korean (n = 330; 0.7%), Japanese (n = 312; 0.7%), and Russian (n = 227; 0.5%). The representation of citations in languages other than English probably reflects both the number of trials published in that language as well as our methods of sourcing the trial citations for the register. For example, the number of trials in Chinese reflects both the fact that Chinese journals publish a large number of trials of traditional Chinese medicine interventions (e.g., acupuncture, Chinese herbal medicine) [23] and also the fact that we have collaborated with our partner institution in China, since 2008, to search Chinese journals and databases to identify these trials. More recent collaborations have resulted in the identification and inclusion in the register of traditional medicine trial citations from other countries (see Table 1). Searches of trials by several of these collaborating institutions are ongoing.

Table 1 Source of citations in the CAM Field specialized register
Full size table

The 25 conventional journal titles and 25 CAM journal titles with the largest number of citations in the register are listed in Tables 2 and 3, respectively. Nearly all (24/25) of the top conventional medicine journals are MEDLINE-indexed, while only 12/25 of the top 25 CAM journals are MEDLINE-indexed. Together, these 50 journals account for 13,731 trials (30.6% of the total) and 8,445 of these trial citations (61.5%) are MEDLINE-indexed. The clinical focus of the conventional journals was concentrated in nutrition (n = 8), general and internal medicine (n = 4), and pediatrics (n = 3). The clinical focus of the top 25 CAM journals was concentrated in TCM (n = 20).

Table 2 Twenty-five conventional medicine journals with the most citations in the CAM Field specialized register
Full size table
Table 3 Twenty-five complementary medicine journals with the most citations in the CAM Field specialized register
Full size table

The CAM register contains citations from 4,845 journals. Citations are quite concentrated in a few journals. One-third of the citations are found in the top 57 journals and two-thirds come from the top 420 journals. Among the 4,425 journals containing the remaining one-third of citations, 2,749/4,425 (62%) contributed only 1 or 2 citations to the register.

Of the 44,840 trial citations in the register, 93% were classified into one or more of the CAM intervention categories for which searches were conducted. The greatest concentrations were in non-vitamin, non-mineral dietary supplements (e.g., glucosamine, fish oil); Chinese herbal medicine (e.g., Astragalus membranaceus, Schisandra chinensis); diet-based therapies; vitamin and mineral interventions; and acupuncture (Table 4). The high representation of acupuncture and Chinese herbal medicine trial citations in the register might be explained by both a large number of trials being published in these areas as well as by the methods that we used to source the trial citations for the database (as described above).

Table 4 Number of CAM Field specialized register citations classified by type of CAM interventions
Full size table

Of the 44,840 citations in the register, 85% were classified into 1 or more of the 25 categories of medical conditions. The greatest concentrations were in the categories of heart and circulation, anesthesia and pain, mental health, and endocrine and metabolic conditions (Table 5). Categories varied greatly in the proportion of citations included on MEDLINE, and the lowest percentage of MEDLINE-indexed citations was among citations not classified into any medical condition category. This is likely a result of non-MEDLINE citations being less likely to have abstracts or detailed keywords, and thus being less easy to categorize through searches.

Table 5 Number of CAM Field specialized register citations classified by type of medical condition
Full size table

Discussion

The CAM Field specialized register is an important resource for both MEDLINE and non-MEDLINE citations of CAM controlled trials. The prevalence of MEDLINE-indexed trial citations reflects the searches conducted in PubMed (and later in CENTRAL) for MEDLINE-indexed citations retrieved using the CAM on PubMed search strategy. We must therefore acknowledge the strides made in identification of both controlled trial and CAM citations by the US National Library of Medicine since the CAM Field specialized register was last examined in 1998 [5]. The large number of non-MEDLINE citations reflects searches of CENTRAL, which includes non-MEDLINE citations, and the efforts of CAM Field collaborators in China, Canada, Japan, and Korea, whose contributions to the specialized register were of predominantly non-MEDLINE citations. Overall, less than two-thirds of register citations are MEDLINE-indexed, and Sampson et al. concluded that with incomplete MEDLINE indexing of a body of literature, a specialized register was of particular utility [24].

Particular strengths of the register include citations of nutritional and supplement-related interventions, and traditional medicine. While citations of chelation therapy, nutrition and supplement-related interventions, sensory art therapies, relaxation, and meditation are likely to be MEDLINE-indexed (at least 80% of all these citations are indexed in MEDLINE), citations of traditional medicine interventions and homeopathy are less likely to be MEDLINE-indexed (fewer than 50% indexed in MEDLINE) and thus may be more difficult to locate. Therefore, the CAM Field specialized register may be a particularly useful resource for identifying citations of trials to be included in systematic reviews of traditional medicine interventions, particularly TCM. Similarly, the CAM Field specialized register may be a useful source of trials for systematic reviews covering CAM interventions for medical conditions in which a lower proportion of citations are MEDLINE-indexed (e.g., endocrine and metabolic conditions or infectious disease).

Among the 4,845 journals containing citations in the CAM Field register, 9% contain two-thirds of the register citations, and 57% contain only one or two register citations each. One unanswered question is whether this distribution of journals in the register represents the true distribution of CAM trials across journals or whether instead it is an artifact resulting from the way that the register was developed. There is no way to definitively answer this question because there exists no ‘gold standard’ complete database of CAM trials against which the journal distribution in the CAM Field register can be compared. However, we believe that the distribution of journals in the CAM Field register is largely an artifact of the way the register was developed. This is because, in identifying citations for register inclusion, contributors often searched bibliographic databases to identify trial citations on specific topics for their systematic reviews rather than comprehensively searching entire journals for all CAM trials. As a result, some journals for which only one or two CAM trial citations were identified for register inclusion may have many additional trials that have not yet been identified. Continued efforts to identify trials will likely change the distribution of citations across journals. Such efforts may also change the characteristics of the register in other ways (e.g., the number and proportion of trials covering particular CAM interventions or published in particular languages) that are not possible to quantify in advance. While the ultimate aim is for the CAM Field register to be a comprehensive source of CAM controlled trials, the register cannot currently be considered to be comprehensive. Therefore, systematic reviewers of CAM interventions should search multiple electronic and other sources for relevant CAM trials, in addition to searching the CAM Field register.

The strength of the register in terms of its coverage of difficult to locate trials may, however, be associated with potential weaknesses in terms of the quality of these trials. The largest subset of non-MEDLINE citations in the register (51%) is trials published in Chinese. These Chinese-language trials were included in the register if the trial publication stated that a random or quasi-random procedure was used to assign participants to treatment groups. However, a recent telephone survey of authors of ‘claimed’ randomized trials conducted in China discovered that only 7% could be confirmed to use a random method to assign participants to treatment groups [25]. Inclusion in systematic reviews of such Chinese trials claiming to be randomized, but not confirmed as such by systematic reviewers, may inflate these reviews’ meta-analytic effect estimates [26]. In addition, a 1998 review of the outcomes of non-English language trials by Vickers et al. found that acupuncture trials conducted in China reported positive results 100% of the time, and Chinese trials of other interventions reported positive results 99% of the time, strongly suggesting the preferential publication in China of trials with positive results [27]. Although Chinese language trials reflect the majority of non-English language trials included in the register, the issue of a publication or reporting bias favoring positive results may also be relevant to other non-English trials included in the register. For example, the Vickers et al. 1998 review found that not only acupuncture trials from China, but also acupuncture trials from Japan, Hong Kong, and Taiwan were uniformly positive. Also, a recent preliminary investigation into the results of the Japanese Kampo trials included in the CAM Field register found that only a small number of Kampo trials have negative results (Kiichiro Tsutani, personal communication, 14 March 2011), and an informal assessment of CAM Korean trials indicated that negative results were rare (Byeung-Cheul Shin, personal communication, 13 July 2012). The positive results of non-English language CAM trials likely explain why including such trials in CAM-related systematic reviews tends to inflate meta-analysis effect estimates, according to a 2005 empirical study [28].

It is possible that the methodological quality of the more recent Chinese-language trials may, however, be better than that of the earlier trials because, for example, the CONSORT statement has recently been more widely disseminated in China [29], including in Chinese journals of TCM [30]. In addition, while the Vickers et al. 1998 review [27] found that Chinese acupuncture trials published up to 1998 were uniformly positive, it is not known whether or not more recent Chinese trials are also likely to be uniformly positive. An important topic area for future research is to determine whether there is a publication bias favoring positive results in more recent trials from China. However, while conducting research studies to assess for the likelihood of publication bias in Chinese trials may be informative in determining the scope of the problem, the only way to avoid publication bias in Chinese trials is to ensure that all initiated Chinese language trials are known about through the registration of Chinese trials at inception, which is currently being implemented [31]. Universal trial registration, in conjunction with reporting of trial registration numbers in publications, might also serve as a tool in addressing duplicate publication [32], which some studies have observed to be prevalent among Chinese, Japanese, and Korean trials [33–35]. In the interim, for systematic reviews including a large number of Chinese trials, a possible approach for assessing the impact of a potential publication bias related to the Chinese trials may be to mark the Chinese trials in funnel plots in systematic reviews.

Despite these concerns over the validity of Chinese trials in general, it seems inappropriate to exclude trials from systematic reviews on the basis of language or country of publication alone. A more measured approach may be to search for Chinese trials and to telephone interview the authors of potentially eligible trials to try and assess whether the trials were truly randomized before including them in the review. If concerns about the validity of the trials remain, even after the telephone interviews with the authors, a possible approach is to include in the review those trials (Chinese or Western) for which there remains uncertainty about whether true randomization was used, but to be more restrictive when presenting the key findings, such as the abstract conclusions and the summary of findings table [36]. Another approach may be to analyze the potential influence of risk of bias measures (e.g., adequacy of randomization) on effect estimates using subgroup analyses or sensitivity analyses. Either way, such assessment and analysis approaches should probably be based on risk of bias measures rather than on the language of country of origin of the trials. This is because generalizing about individual Chinese language trials, for example, based on meta-research of the characteristics of Chinese trials overall would be an erroneous oversimplification. Instead, each trial included in a systematic review needs to be individually evaluated on its risks of bias, assuming either that the trial publication is sufficiently informative or that the trial author can be contacted for further information. Such risk of bias assessments can then be incorporated into the review’s analysis.

In addition to providing a source of trials for inclusion in systematic reviews, the register may also be used for investigations into the optimal use of CAM research resources and the prioritization of future CAM reviews. This analysis of the types of CAM interventions and health conditions covered in the register is a first step in conducting such investigations. That is, the number of citations related to various CAM interventions (e.g., diet-based therapies) and the number of citations related to various medical conditions (e.g., endocrine and metabolic disorders) may be used for research into whether there is a correlation between those CAM interventions most frequently investigated in trials and those most commonly used, and whether the most serious or prevalent health conditions are proportionately represented with the highest number of CAM trials. If trials of commonly used interventions and/or trials for serious or prevalent health conditions are lacking, this may indicate that CAM research resources should be directed to these areas. In addition, the trial database may also be useful for prioritizing future Cochrane reviews by identifying CAM intervention/health condition pairings for which there are available trials in the register but no existing Cochrane review. The fact that the same classification categories were used for CAM intervention types and health conditions, for both the trial citations in the specialized register and for a separate database of CAM-related Cochrane reviews [10], should facilitate such identification. However identifying potential future systematic reviews to prepare will require additional narrowing down of some of our CAM intervention type categories (e.g., “Chinese herbal medicine”) and health condition categories (e.g., “mental health”) in order to identify more specific intervention/condition pairings (e.g., the Chinese herbal medicine formula Free and Easy Wanderer for depression) for systematic reviews. Future plans for the CAM Field specialized register include augmenting the size and scope of the register through ongoing searches and international partnerships, and developing methods to characterize groups of trials according to intervention/condition characteristics and mapping these groups to gaps in Cochrane systematic reviews.

Conclusions

The number of citations included in the CAM Field specialized register increased nearly tenfold between 2006 and 2011 as a result of a program of extensive searching and partnerships with international collaborators. Many CAM Field register citations are not MEDLINE-indexed and many of these non-MEDLINE-indexed citations are published in languages other than English. While the register provides access to thousands of difficult to locate citations of trials, many of these trials are likely to be of low quality and may overestimate treatment effects. When these trials are considered for inclusion in systematic reviews, it is extremely important that their risk of bias is adequately assessed.

References

  1. Dickersin K, Manheimer E, Wieland S, Robinson KA, Lefebvre C, McDonald S: Development of the Cochrane Collaboration's CENTRAL Register of controlled clinical trials. Eval Health Prof. 2002, 25: 38-64.

    Article  PubMed  Google Scholar 

  2. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J: How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003, 7: 1-76.

    CAS  PubMed  Google Scholar 

  3. Hofmans EA: Acupuncture and Medline. Lancet. 1990, 336: 57-

    Article  CAS  PubMed  Google Scholar 

  4. Kleijnen J, Knipschild P: The comprehensiveness of Medline and Embase computer searches. Searches for controlled trials of homoeopathy, ascorbic acid for common cold and ginkgo biloba for cerebral insufficiency and intermittent claudication. Pharm Weekbl Sci. 1992, 14: 316-320. 10.1007/BF01977620.

    Article  CAS  PubMed  Google Scholar 

  5. Vickers AJ: Bibliometric analysis of randomized trials in complementary medicine. Complement Ther Med. 1998, 6: 185-189. 10.1016/S0965-2299(98)80026-5.

    Article  Google Scholar 

  6. Lefebvre C, Manheimer E, Glanville J: Chapter 6: Searching for studies. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration, Available from http://www.cochrane-handbook.org (accessed 14 January 2013)

    Google Scholar 

  7. Committee on the Use of Complementary and Alternative Medicine by the American Public: Chapter 5: State of emerging evidence on CAM. Board of Health Promotion and Disease Prevention, Institute of Medicine of the National Academies. Complementary and Alternative Medicine in the United States. 2005, Washington DC: The National Academies Press, Available from http://books.nap.edu/catalog/11182.html (accessed 14 January 2013)

    Google Scholar 

  8. US National Center for Complementary and Alternative Medicine: About CAM on PubMed. Bethesda, MD: US Department of Health and Human Services. National Institutes of Health, Available from http://nccam.nih.gov/research/camonpubmed (accessed 14 January 2013)

  9. US National Library of Medicine: Search strategy used to create the Complementary Medicine subset on PubMed. Bethesda, MD: US Department of Health and Human Services. National Institutes of Health, Available from http://www.nlm.nih.gov/bsd/pubmed_subsets/comp_med_strategy.html (accessed 14 January 2013)

  10. Wieland LS, Manheimer E, Berman BM: Development and classification of an operational definition of complementary and alternative medicine for the Cochrane collaboration. Altern Ther Health Med. 2011, 17: 50-59. *Both authors contributed equally to this work

    PubMed  PubMed Central  Google Scholar 

  11. World Health Organization: Traditional medicine, Fact Sheet No. 134. 2008, Geneva: World Health Organization, Available from http://www.who.int/mediacentre/factsheets/fs134/en/index.html (accessed 14 January 2013)

    Google Scholar 

  12. Cogo E, Sampson M, Ajiferuke I, Manheimer E, Campbell K, Daniel R, Moher D: Searching for controlled trials of complementary and alternative medicine: a comparison of 15 databases. Evid Based Complement Alternat Med. 2011, 2011: 858246-Epub 2011 Jun 23

    Article  PubMed  PubMed Central  Google Scholar 

  13. Kim KH, Kong JC, Choi JY, Choi TY, Shin BC, McDonald S, Lee MS: Impact of including Korean randomized controlled trials in Cochrane reviews of acupuncture. PLoS One. 2012, 7: e47619-10.1371/journal.pone.0047619.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Okabe T, Tsutani K: Evidence Reports of Kampo Treatment 2010:345 Randomized Controlled Trials (EKAT 2010). 2011, Task Force for Evidence Reports / Clinical Practice Guidelines (ER/CPG-TF), Special Committee for Evidence-based Medicine (EBM), The Japan Society for Oriental Medicine (JSOM), Available from http://www.jsom.or.jp/medical/ebm/ere/pdf.html (accessed 14 January 2013)

    Google Scholar 

  15. Barnabas J, Yamuna BGC, Parthasarathy V, Venkatesh S, Tharyan P: 17th International Cochrane Colloquium; 2009 Oct 11–14. Access to evidence from countries in South Asia: the South Asian Database of Controlled Clinical Trials and the South Asian Cochrane Network and Centre’s Digital Library. Singapore, Abstract no. P06-8. Available at http://www.imbi.uni-freiburg.de/OJS/cca/index.php?journal=cca&page=article&op=view&path%5B%5D=8010 (accessed 14 January 2013)

  16. Pienaar E: 13th International Cochrane Colloquium; 2005 Oct 22–26. African Trials Register: 5 years later, where are we now. Melbourne, Abstract no. P021. Available at http://www.imbi.uni-freiburg.de/OJS/cca/index.php?journal=cca&page=article&op=view&path%5B%5D=1232 (accessed 14 January 2013)

  17. Siegfried N, Busgeeth K, Clarke M, Pienaar E, Volmink L: 14th International Cochrane Colloquium; 2006 October 23–26. African randomized controlled trials - too hard to find. Dublin, Abstract no. P017. Available at http://www.imbi.uni-freiburg.de/OJS/cca/index.php?journal=cca&page=article&op=view&path&%5B%5D=1972 (accessed 14 January 2013)

  18. Barnes PM, Bloom B, Nahin RL: Complementary and alternative medicine use among adults and children: United States. Natl Health Stat Report. 2008, 2007: 1-23.

    Google Scholar 

  19. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C: An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009, 62: 944-952. 10.1016/j.jclinepi.2008.10.012.

    Article  PubMed  Google Scholar 

  20. The Cochrane Library: Independent high quality evidence for healthcare decision making. John Wiley & Sons, Ltd, Available from http://www.thecochranelibrary.com (accessed 14 January 2013)

  21. Cochrane Central Register of Controlled Trials (CENTRAL): Issue 1, part of The Cochrane Library. 2012,http://www.thecochranelibrary.com,

    Google Scholar 

  22. Clarke M, Clarke T: A project funded in part by a grant from The Cochrane Collaboration’s Discretionary Fund. The Cochrane Central Register of Controlled Trials and the contribution from the specialized registers of Cochrane Review Groups. 2008

    Google Scholar 

  23. Tang JL, Zhan SY, Ernst E: Review of randomised controlled trials of traditional Chinese medicine. BMJ. 1999, 319: 160-161. 10.1136/bmj.319.7203.160.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Sampson M, Daniel R, Cogo E, Dingwall O: Sources of evidence to support systematic reviews in librarianship. J Med Libr Assoc. 2008, 96: 66-69. 10.3163/1536-5050.96.1.66.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Wu T, Li Y, Bian Z, Liu G, Moher D: Randomized trials published in some Chinese journals: how many are randomized?. Trials. 2009, 10: 46-10.1186/1745-6215-10-46.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL: Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012, 157: 429-438.

    Article  PubMed  Google Scholar 

  27. Vickers A, Goyal N, Harland R, Rees R: Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998, 19: 159-166. 10.1016/S0197-2456(97)00150-5.

    Article  CAS  PubMed  Google Scholar 

  28. Pham B, Klassen TP, Lawson ML, Moher D: Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005, 58: 769-776. 10.1016/j.jclinepi.2004.08.021.

    Article  PubMed  Google Scholar 

  29. MacPherson H, Altman DG: Improving the quality of reporting acupuncture interventions: describing the collaboration between STRICTA, CONSORT and the Chinese Cochrane Centre. J Evid Based Med. 2009, 2: 57-60. 10.1111/j.1756-5391.2009.01008.x.

    Article  PubMed  Google Scholar 

  30. Fei YT, Liu JP: Improving the quality of reporting Chinese herbal medicine trials: an elaborated checklist. J Chinese Integr Med (Zhong Xi Yi Jie He Xue Bao). 2008, 6: 233-238. 10.3736/jcim20080302.

    Article  Google Scholar 

  31. Wu T, Li Y, Liu G, Li J, Wang L, Du L: Chinese clinical trial registry: mission, responsibility and operation. J Evid Based Med. 2011, onlinelibrary.wiley.com/doi/10.1111/j.1756-5391.2011.01137.x/pdf. [ Epub ahead of print]

    Google Scholar 

  32. Antes G, Dickersin K: Trial registration to prevent duplicate publication. JAMA. 2004, 291: 2432-

    CAS  PubMed  Google Scholar 

  33. Errami M, Garner H: A tale of two citations. Nature. 2008, 451: 397-399. 10.1038/451397a.

    Article  CAS  PubMed  Google Scholar 

  34. Kitagawa M, Tsutani K: Duplicate publication cases in the field of Kampo (Japanese herbal medicine) in Japan. J Chinese Integr Med (Zhong Xi Yi Jie He Xue Bao). 2011, 9: 1055-1060. 10.3736/jcim20111003.

    Article  Google Scholar 

  35. Kim SY, Hahm CK, Bae CW, Cho HM: Duplicate publications in Korean medical journals indexed in Korea Med. J Korean Med Sci. 2008, 23: 131-133. 10.3346/jkms.2008.23.1.131.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Deeks JJ, Higgins JPT, Altman DG: Chapter 9: Analysing data and undertaking meta-analyses. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration, Available from http://www.cochrane-handbook.org (accessed 14 January 2013)

    Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the assistance of Meizi Zheng and Albra Prager of the Center for Integrative Medicine at the University of Maryland in downloading trial citations and checking and conforming journal titles for this project. We would like to acknowledge Tamara Rader, MLIS, Cochrane Musculoskeletal Group at the University of Ottawa, for assistance with the classification of trials by medical condition and type of CAM intervention.

Guarantor of the article: L. Susan Wieland, MPH, PhD.

Financial support

L. Susan Wieland, Eric Manheimer, and Brian M. Berman were partially funded by grant no. R24 AT001293 from the National Center for Complementary and Alternative Medicine (NCCAM) of the US National Institutes of Health. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the NCCAM or the National Institutes of Health. Myeong Soo Lee was supported by Korea Institute of Oriental Medicine (K13400). Jianping Liu and Xun Li were supported by the Program for Innovative Research Team of Beijing University of Chinese Medicine (no. 2011-CXTD-09) and a grant from Basic Operational Funding for Scientific Research from Beijing University of Chinese Medicine. David Moher was funded through a University of Ottawa Research Chair and the Canadian Institutes of Health Research.

Author information

Authors and Affiliations

  1. University of Maryland School of Medicine, Baltimore, MD, USA

    L Susan Wieland, Eric Manheimer & Brian M Berman

  2. Children’s Hospital of Eastern Ontario, Ottawa, Canada

    Margaret Sampson

  3. Christian Medical College, Vellore, India

    Jabez Paul Barnabas & Prathap Tharyan

  4. VU University Amsterdam, Amsterdam, The Netherlands

    Lex M Bouter

  5. Kyunghee University, Seoul, South Korea

    Kiho Cho

  6. Korea Institute of Oriental Medicine, Seoul, South Korea

    Myeong Soo Lee

  7. Beijing University of Chinese Medicine, Beijing, China

    Xun Li & Jianping Liu

  8. Ottawa Hospital Research Institute, University of Ottawa, Ottawa, Canada

    David Moher

  9. University of Tokyo, Tokyo, Japan

    Tetsuro Okabe & Kiichiro Tsutani

  10. South African Cochrane Centre, South African Medical Research Council, Cape Town, South Africa

    Elizabeth D Pienaar

  11. Pusan National University, Busan, South Korea

    Byung-Cheul Shin

  12. Keele University, Staffordshire, ST5 5BG, UK

    Daniëlle A van der Windt

Authors
  1. L Susan Wieland
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Manheimer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Margaret Sampson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jabez Paul Barnabas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lex M Bouter
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kiho Cho
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Myeong Soo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jianping Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. David Moher
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Tetsuro Okabe
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Elizabeth D Pienaar
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Byung-Cheul Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Prathap Tharyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Kiichiro Tsutani
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Daniëlle A van der Windt
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Brian M Berman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to L Susan Wieland.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

Study concept and design: LSW, EM. Contribution of trial citations: JPB (citations of trials of Ayurveda and other CAM interventions included in the South Asian Database of Controlled Clinical Trials), KC (citations of Korean controlled trials of traditional medicine interventions), MSL (citations of Korean controlled trials of traditional medicine interventions), XL (citations of Chinese controlled trials of TCM interventions), JL (citations of Chinese controlled trials of TCM interventions), DM (citations of CAM controlled trials identified from specialized databases), TO (citations of Kampo controlled trials identified and evaluated by the Japan Society for Oriental Medicine), EP (citations of CAM-related controlled trials included in the African Trials Register), MS (citations of CAM controlled trials identified from specialized databases), B-CS (citations of Korean controlled trials of traditional medicine interventions), PT (citations of trials of Ayurveda and other CAM interventions included in the South Asian Database of Controlled Clinical Trials), KT (citations of Kampo controlled trials identified and evaluated by the Japan Society for Oriental Medicine). Data management: LSW, EM. Analysis of data: LSW, EM. Interpretation of data: EM, LSW, MS, DAvdW, LMB. Drafting the manuscript: Introduction, Methods, and Results: LSW, EM. Discussion: EM, LSW. Critically revised manuscript for important intellectual content and provided approval of the final manuscript: EM, LSW, MS, JPB, BMB, LMB, KC, MSL, XL, JL, DM, TO, EP, B-CS, PT, KT, and DAvdW. All authors read and approved the final manuscript.

Electronic supplementary material

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Wieland, L.S., Manheimer, E., Sampson, M. et al. Bibliometric and content analysis of the Cochrane Complementary Medicine Field specialized register of controlled trials. Syst Rev 2, 51 (2013). https://doi.org/10.1186/2046-4053-2-51

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/2046-4053-2-51

Keywords

Systematic Reviews

ISSN: 2046-4053