Quality of reporting of systematic reviews published in “evidence-based” Chinese journals

Background The number of systematic reviews (SRs)/meta-analyses (MAs) has increased dramatically in China over the past decades. However, evaluation of quality of reporting of systematic reviews published has not been undertaken. The objective of this study is to evaluate the quality of reporting of SRs/MAs assessing efficacy and/or harms of clinical interventions published in “evidence-based” Chinese journals. Methods Web-based database searches were conducted for the Chinese Journal of Evidence-based Medicine, the Journal of Evidence-Based Medicine, the Chinese Journal of Evidence Based Pediatrics, and the Chinese Journal of Evidence-Based Cardiovascular Medicine. SRs/MAs assessing efficacy and/or harms of clinical interventions were included. The cut-off was December 31st 2011. The PRISMA statement was applied to assess the quality of reporting. Each item was assessed as follows: ‘Yes’ for total compliance, scored ‘1’; ‘partial’ for partial compliance, scored ‘0.5’; and ‘No’ for non-compliance, scored ‘0’. The review was considered to have major flaws if it received a total score of ≤15.0, minor flaws if it received a total score of 15.5 to 21.0, and minimal flaws if it received a total score 21.5 to 27.0. Odds ratios were used for binary variables, and the mean difference was used for continuous variables. Analyses were performed using RevMan 5.0 software. Results Overall, 487 SRs/MAs were identified and assessed. The included reviews had medium quality with minor flaws based on PRISMA total scores (range: 8.5–26.0; mean: 19.6 ± 3.3). The stratified analysis showed that SRs/MAs with more than 3 authors, from a university, hospital + university cooperation, multiple affiliations (≥2), and funding have significantly higher quality of reporting of SRs/MAs; 58% of the included reviews were considered to have minor flaws (total score of 15.6 to 21.0). Only 9.6% of reviews were considered to have major flaws. Specific areas needing improvement in reporting include the abstract, protocol and registration, and characteristics of the search. Conclusions The reporting of SRs published in “evidence-based” Chinese journals is poor and needs to be improved in order for reviews to be useful. SR authors should use the PRISMA checklist to ensure complete and accurate accounts of their SRs.


Background
Clinicians refer to systematic reviews (SRs)/meta-analyses (MAs) to keep up to date with developments in their field. High quality SRs/MAs of randomized controlled trials can provide the best evidence [1]. In order to assess the quality of reporting of SRs/MAs, complete, clear, and transparent information with regards to the design and conduct of the SRs/MAs are required; however, to date, this information it is not optimal and the low quality of reporting diminishes the value of SRs/MAs for clinicians, policy makers, and other users [2].
In order to improve the quality of reporting of SRs/MAs, an international group of experienced authors and methodologists developed the Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA) [3], which itself was a successor to the original QUOROM guidelines [4]. It contains a 27-item checklist and a 4-phase flow diagram. The checklist includes items deemed essential for transparent reporting of a SR [2].
A few studies have examined the reporting quality of SRs/MAs in various fields, and found that the reporting quality of SRs/MAs was poor with many flaws [5][6][7][8].
Recently, the concept of evidence-based medicine has widely spread across China through the Chinese Journal of Evidence-based Medicine (CJEBM), the Journal of Evidence-Based Medicine (JEBM), the Chinese Journal of Evidence-Based Pediatrics (CJEBP), and the Chinese Journal of Evidence-Based Cardiovascular Medicine (EBCVM) [9]. In this study, we sought to determine the degree to which studies published in these four journals met the PRISMA criteria for reporting, whether it had improved after publication of the PRISMA statement, and factors associated with good reporting.

Methods
Sample collection, selection, data extraction, and quality assessment were performed independently by two trained reviewers; disagreements between reviewers were resolved through consensus or by consulting a third expert adjudicator.

Selection of studies
An inclusion criterion was that the words SR and/or MA were stated in the title of studies. Using this criterion, we searched the online databases of the CJEBM (www.cjebm.org.cn, 2001 to December 31st, 2011), the JEBM (www.jebm.cn, 2001 to December 31st, 2011), the CJEBP (www.cjebp.net, 2006 to December 31st, 2011) and the EBCVM (www.ebcvm.org, 2008 to December 31st, 2011) for SRs and MAs. Only reviews assessing the safety and/or efficacy of clinical interventions were included. A total of 487 SRs and MAs on interventions were included after reading the title, abstract, and full-text.

Data extraction
We established a spreadsheet using Microsoft Excel 2003. In addition to the items in PRISMA, the following data were extracted from each study: i) title; ii) first author name; iii) the name of the journal; iv) the number of author(s); v) year of publication; vi) the authors' affiliations (hospital or university) and the number of affiliations; vii) financial support; viii) categories of diseases (ICD-10) [10]; and ix) the number of included studies.

Statistical analysis
PRISMA was used to evaluate the adherence of review articles to scientific principles. The tool covers seven modules with 27-items: title, abstract, introduction, methods, results, discussion, and funding. Each of the items were assessed as follows: 'Yes' for total compliance, scored '1'; 'partial' for partial compliance, scored '0.5'; and 'No' for non-compliance, scored '0'; with a total maximum score '27'. The review was considered to have major flaws if it received a total score of ≤15.0, minor flaws if it received a total score of 15.5 to 21.0, and minimal flaws if it received a total score 21.5 to 27.0 [11].
The percentage of reports that met each criterion was determined and tabulated. Data on each item was presented as counts and percentages. The odds ratio (OR), 95% CI, mean, standard deviation (SD), and mean difference (MD) were used as the summary statistics for stratified comparisons with RevMan software version 5.0. Statistical significance was set at P ≤0.05.

Stratified analysis
We assessed the following variables as potentially associated with differences in the quality of reporting: publication time (≤2009 vs. ≥2010), the number of authors (≤2 vs. ≥3), the authors' affiliations (hospital vs. university), the number of affiliations (1 vs. ≥2), and funding sources (funding vs. non-funding) [12].
To assess whether publication of the PRISMA statement was associated with improved reporting of certain items, we compared overall compliance in each PRISMA item using the t-test (≤2009 vs. ≥2010).

Description of included reviews
Four hundred and eighty seven studies were included for evaluation; 77.8% (379/487) of SRs/MAs were published in CJEBM, 14.4% (70/487) of SRs were published in JEBM, 6.2% (30/487) of SRs were published in CJEBP, and EBCVM published 1.6% (8/487) of SRs/MAs. The first SR was published in 2001, and the total number increased to 89 in 2009. Twenty categories of diseases were involved according to ICD-10 and the most common conditions studied were neoplasms (17.4%, 83/487). The reviews included a median of four authors (range: 1-12) and 87.7% (427/487) of the studies were written by ≥3 authors; almost half (43.3%) of the reviews were written by clinicians. Financial support was obtained by 25% (122/487) of the reviews. The reviews included a median of 8 randomized controlled trials (range: 1-129).
The overall reporting quality of included reviews Overall, the compliance with PRISMA was poor, none of the included reviews fulfilled all 27-items of PRISMA. Table 1 shows that the weakest areas within the included SRs/MAs were the reporting of protocol and registration information (item 5, 0.4%), risk of bias across studies (item 22, 22.4%), additional analyses (item 23, 29.2%), and funding sources (item 27, 24.4%). The reasons for particularly low reporting for item 5 may be that there is no existing register for SRs/MAs in China, and the relatively rare publication of SR/MA protocols. Some PRISMA  To investigate whether the publication of the PRISMA statement was associated with an improvement in reporting for certain items (pre-PRISMA vs. post-PRISMA), the period ≤2009 was compared with ≥2010 in each of the PRISMA statement. Figure 2 shows that, following publication of the PRISMA statement, the quality of reporting improved; the difference was statistically significant for following items: item 1 (P = 0.0006), item 6 (P = 0.0006), item 8 (P = 0.04), item 9 (P <0.0001), item 11 (P <0.00001), and item 14 (P = 0.0002).

Stratified analysis
To assess possible factors associated with reporting quality, we analyzed the publication time (≤2009 vs. ≥2010), the number of authors (≤2 vs. ≥3), the affiliation of authors  Comparison of overall PRISMA scores in the four journals assessed Table 2 shows that the overall quality score of SRs/MAs published in the four journals demonstrated minor flaws (58.3%), and we found that the quality of reporting of SRs/MAs published in CJEBP (21.8 ± 1.5) was better when compared with the remaining journals (19.5 ± 3.4) (MD: 2.3 [95% CI: 1.7 to 2.9], P <0.00001).

Discussion
The number of SRs/MAs published in China has been accumulating at an increasing rate in recent years. However, the results of our study indicate that the quality of the SRs/MAs is not optimal. The best reported items (addressed completely >90.0%) included identification of a SR/MA by its title, eligibility criteria, information sources (such as the database or key words searched), summary measures, synthesis of results (methods section), study characteristics, results of individual studies (results section), synthesis of results (results section), and summary of evidence (results section). A great deal of improvement is required in the following items (addressed completely 50% to 90%): structured summary, rational, search, data collection process, risk of bias in individual studies (methods section), additional analyses (methods section), study selection, risk of bias with studies (results section), limitations, and conclusions. Furthermore, there was seriously flawed reporting (addressed completely 50% or less) in protocol and registration, study selection (methods section), data items (methods section), risk of bias across studies (methods section), risk of bias across studies (results section), additional analyses (results section), and funding sources.
Further, the registration of protocols was rarely performed. We expect that a network of registration for SRs will be established in China, thus facilitating the registration and reporting of protocols. The main reason most studies scored poorly on details of the search strategy was that only 19.7% reviews reported the detailed search strategy; most reviews only reported the databases and key words. Chinese journals should demand that authors provide the detailed search strategy.

Conclusions
SRs or MAs related to efficacy and safety of clinical interventions published in "evidence-based" Chinese journals had minor flaws. Efforts should be made to improve the reporting of a structured abstract, objectives, protocol and registration, search strategy, data extracted, risk of bias across studies, additional analyses, and funding sources reporting. Meanwhile, SR authors should use the PRISMA checklist to ensure complete and accurate accounts of their SRs.

Competing interests
The authors declare that they have no competing interest.
Authors' contributions TJH and GL have participated in all study processes, including the concept and design, analysis and interpretation of data, drafting and revising of the manuscript, and approving the manuscript. LJL has participated in analysis and interpretation of data, and written the manuscript. DJX, GYH, YL, and AN have participated in literature search, data extraction, and quality assessment. MJC and ZQL have participated in statistical analysis and drafting and revising of the manuscript. All authors read and approved the final manuscript.