Quality of reporting of systematic reviews published in “evidence-based” Chinese journals
- Jin-long Li†1, 2,
- Long Ge†1, 2,
- Ji-chun Ma1, 2,
- Qiao-ling Zeng1, 3,
- Lu Yao1, 3,
- Ni An1, 2,
- Jie-xian Ding1, 3,
- Yu-hong Gan1 and
- Jin-hui Tian1Email author
© Li et al.; licensee BioMed Central Ltd. 2014
Received: 26 November 2013
Accepted: 29 May 2014
Published: 7 June 2014
The number of systematic reviews (SRs)/meta-analyses (MAs) has increased dramatically in China over the past decades. However, evaluation of quality of reporting of systematic reviews published has not been undertaken. The objective of this study is to evaluate the quality of reporting of SRs/MAs assessing efficacy and/or harms of clinical interventions published in “evidence-based” Chinese journals.
Web-based database searches were conducted for the Chinese Journal of Evidence-based Medicine, the Journal of Evidence-Based Medicine, the Chinese Journal of Evidence Based Pediatrics, and the Chinese Journal of Evidence-Based Cardiovascular Medicine. SRs/MAs assessing efficacy and/or harms of clinical interventions were included. The cut-off was December 31st 2011. The PRISMA statement was applied to assess the quality of reporting. Each item was assessed as follows: ‘Yes’ for total compliance, scored ‘1’; ‘partial’ for partial compliance, scored ‘0.5’; and ‘No’ for non-compliance, scored ‘0’. The review was considered to have major flaws if it received a total score of ≤15.0, minor flaws if it received a total score of 15.5 to 21.0, and minimal flaws if it received a total score 21.5 to 27.0. Odds ratios were used for binary variables, and the mean difference was used for continuous variables. Analyses were performed using RevMan 5.0 software.
Overall, 487 SRs/MAs were identified and assessed. The included reviews had medium quality with minor flaws based on PRISMA total scores (range: 8.5–26.0; mean: 19.6 ± 3.3). The stratified analysis showed that SRs/MAs with more than 3 authors, from a university, hospital + university cooperation, multiple affiliations (≥2), and funding have significantly higher quality of reporting of SRs/MAs; 58% of the included reviews were considered to have minor flaws (total score of 15.6 to 21.0). Only 9.6% of reviews were considered to have major flaws. Specific areas needing improvement in reporting include the abstract, protocol and registration, and characteristics of the search.
The reporting of SRs published in “evidence-based” Chinese journals is poor and needs to be improved in order for reviews to be useful. SR authors should use the PRISMA checklist to ensure complete and accurate accounts of their SRs.
KeywordsChinese journal Evidence-based Meta-analyses Quality of reporting Systematic reviews
Clinicians refer to systematic reviews (SRs)/meta-analyses (MAs) to keep up to date with developments in their field. High quality SRs/MAs of randomized controlled trials can provide the best evidence. In order to assess the quality of reporting of SRs/MAs, complete, clear, and transparent information with regards to the design and conduct of the SRs/MAs are required; however, to date, this information it is not optimal and the low quality of reporting diminishes the value of SRs/MAs for clinicians, policy makers, and other users.
In order to improve the quality of reporting of SRs/MAs, an international group of experienced authors and methodologists developed the Preferred Reporting Items of Systematic Reviews and Meta-Analyses (PRISMA), which itself was a successor to the original QUOROM guidelines. It contains a 27-item checklist and a 4-phase flow diagram. The checklist includes items deemed essential for transparent reporting of a SR.
A few studies have examined the reporting quality of SRs/MAs in various fields, and found that the reporting quality of SRs/MAs was poor with many flaws[5–8]. Recently, the concept of evidence-based medicine has widely spread across China through the Chinese Journal of Evidence-based Medicine (CJEBM), the Journal of Evidence-Based Medicine (JEBM), the Chinese Journal of Evidence-Based Pediatrics (CJEBP), and the Chinese Journal of Evidence-Based Cardiovascular Medicine (EBCVM). In this study, we sought to determine the degree to which studies published in these four journals met the PRISMA criteria for reporting, whether it had improved after publication of the PRISMA statement, and factors associated with good reporting.
Sample collection, selection, data extraction, and quality assessment were performed independently by two trained reviewers; disagreements between reviewers were resolved through consensus or by consulting a third expert adjudicator.
Selection of studies
An inclusion criterion was that the words SR and/or MA were stated in the title of studies. Using this criterion, we searched the online databases of the CJEBM (http://www.cjebm.org.cn, 2001 to December 31st, 2011), the JEBM (http://www.jebm.cn, 2001 to December 31st, 2011), the CJEBP (http://www.cjebp.net, 2006 to December 31st, 2011) and the EBCVM (http://www.ebcvm.org, 2008 to December 31st, 2011) for SRs and MAs. Only reviews assessing the safety and/or efficacy of clinical interventions were included. A total of 487 SRs and MAs on interventions were included after reading the title, abstract, and full-text.
We established a spreadsheet using Microsoft Excel 2003. In addition to the items in PRISMA, the following data were extracted from each study: i) title; ii) first author name; iii) the name of the journal; iv) the number of author(s); v) year of publication; vi) the authors’ affiliations (hospital or university) and the number of affiliations; vii) financial support; viii) categories of diseases (ICD-10); and ix) the number of included studies.
PRISMA was used to evaluate the adherence of review articles to scientific principles. The tool covers seven modules with 27-items: title, abstract, introduction, methods, results, discussion, and funding. Each of the items were assessed as follows: ‘Yes’ for total compliance, scored ‘1’; ‘partial’ for partial compliance, scored ‘0.5’; and ‘No’ for non-compliance, scored ‘0’; with a total maximum score ‘27’. The review was considered to have major flaws if it received a total score of ≤15.0, minor flaws if it received a total score of 15.5 to 21.0, and minimal flaws if it received a total score 21.5 to 27.0.
The percentage of reports that met each criterion was determined and tabulated. Data on each item was presented as counts and percentages. The odds ratio (OR), 95% CI, mean, standard deviation (SD), and mean difference (MD) were used as the summary statistics for stratified comparisons with RevMan software version 5.0. Statistical significance was set at P ≤0.05.
We assessed the following variables as potentially associated with differences in the quality of reporting: publication time (≤2009 vs. ≥2010), the number of authors (≤2 vs. ≥3), the authors’ affiliations (hospital vs. university), the number of affiliations (1 vs. ≥2), and funding sources (funding vs. non-funding).
To assess whether publication of the PRISMA statement was associated with improved reporting of certain items, we compared overall compliance in each PRISMA item using the t-test (≤2009 vs. ≥2010).
Description of included reviews
Four hundred and eighty seven studies were included for evaluation; 77.8% (379/487) of SRs/MAs were published in CJEBM, 14.4% (70/487) of SRs were published in JEBM, 6.2% (30/487) of SRs were published in CJEBP, and EBCVM published 1.6% (8/487) of SRs/MAs. The first SR was published in 2001, and the total number increased to 89 in 2009. Twenty categories of diseases were involved according to ICD-10 and the most common conditions studied were neoplasms (17.4%, 83/487). The reviews included a median of four authors (range: 1–12) and 87.7% (427/487) of the studies were written by ≥3 authors; almost half (43.3%) of the reviews were written by clinicians. Financial support was obtained by 25% (122/487) of the reviews. The reviews included a median of 8 randomized controlled trials (range: 1–129).
The overall reporting quality of included reviews
The results of reporting quality assessment (n = 487)
2. Structured summary
5. Protocol and registration
6. Eligibility criteria
7. Information sources
9. Study selection
10. Data collection process
11. Data items
12. Risk of bias in individual studies
13. Summary measures
14. Synthesis of results
15. Risk of bias across studies
16. Additional analyses
17. Study selection
18. Study characteristics
19. Risk of bias with studies
20. Results of individual studies
21. Synthesis of results
22. Risk of bias across studies
23. Additional analyses
24. Summary of evidence
19.60 ± 3.33
Did the quality of SRs/MAs improve post-PRISMA?
Comparison of overall PRISMA scores in the four journals assessed
Comparison of overall PRISMA scores in the four assessed journals [n (%)]
CJEBM (n = 379)
JEBM (n = 70)
CJEBP (n = 30)
EBCVM (n = 8)
Total (n = 487)
Mean ± SD
19.97 ± 3.15
16.60 ± 3.22
21.77 ± 1.52
20.19 ± 1.41
19.60 ± 3.33
The number of SRs/MAs published in China has been accumulating at an increasing rate in recent years. However, the results of our study indicate that the quality of the SRs/MAs is not optimal.
The best reported items (addressed completely >90.0%) included identification of a SR/MA by its title, eligibility criteria, information sources (such as the database or key words searched), summary measures, synthesis of results (methods section), study characteristics, results of individual studies (results section), synthesis of results (results section), and summary of evidence (results section). A great deal of improvement is required in the following items (addressed completely 50% to 90%): structured summary, rational, search, data collection process, risk of bias in individual studies (methods section), additional analyses (methods section), study selection, risk of bias with studies (results section), limitations, and conclusions. Furthermore, there was seriously flawed reporting (addressed completely 50% or less) in protocol and registration, study selection (methods section), data items (methods section), risk of bias across studies (methods section), risk of bias across studies (results section), additional analyses (results section), and funding sources.
Further, the registration of protocols was rarely performed. We expect that a network of registration for SRs will be established in China, thus facilitating the registration and reporting of protocols. The main reason most studies scored poorly on details of the search strategy was that only 19.7% reviews reported the detailed search strategy; most reviews only reported the databases and key words. Chinese journals should demand that authors provide the detailed search strategy.
SRs or MAs related to efficacy and safety of clinical interventions published in “evidence-based” Chinese journals had minor flaws. Efforts should be made to improve the reporting of a structured abstract, objectives, protocol and registration, search strategy, data extracted, risk of bias across studies, additional analyses, and funding sources reporting. Meanwhile, SR authors should use the PRISMA checklist to ensure complete and accurate accounts of their SRs.
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Chinese Journal of Evidence-based Medicine
Journal of Evidence-Based Medicine
Chinese Journal of Evidence Based Pediatrics
Chinese Journal of Evidence-Based Cardiovascular Medicine
Theoretical support was given by Evidence Based Medical Center of Lanzhou University. Thanks teacher Jin-Hui Tian and all the colleagues for their help on this work.
No current external funding sources for this study.
- Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, Cook RJ: Users’ guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. JAMA. 1995, 274: 1800-1804. 10.1001/jama.1995.03530220066035.View ArticlePubMedGoogle Scholar
- Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009, 62: e1-e34. 10.1016/j.jclinepi.2009.06.006.View ArticlePubMedGoogle Scholar
- Moher D, Liberati A, Tetzlaff J, Altman DG: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surq. 2010, 8 (5): 336-341. 10.1016/j.ijsu.2010.02.007.View ArticleGoogle Scholar
- Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Rev Esp Salud Publica. 2000, 74 (2): 107-118.View ArticlePubMedGoogle Scholar
- Jadad AR, McQuay HJ: Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. Clin Epidemiol. 1996, 49 (2): 235-243. 10.1016/0895-4356(95)00062-3.View ArticleGoogle Scholar
- Petticrew M: Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ. 2000, 322: 98-101.View ArticleGoogle Scholar
- Willis BH, Quigley M: The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review. J BMC Med Res Methodol. 2011, 11: 163-10.1186/1471-2288-11-163.View ArticleGoogle Scholar
- Ma B, Guo J, Qi G, Li H, Peng J, Zhang Y, Ding Y, Yang K: Quality and reporting characteristics of systematic reviews of traditional Chinese medicine interventions published in Chinese journals. PLoS One. 2011, 6 (5): e20185-10.1371/journal.pone.0020185.View ArticlePubMedPubMed CentralGoogle Scholar
- Shen J, Yao L, Li Y, Clarke M, Gan Q, Li Y, Fan Y, Gou Y, Wang L: Visualization Studies on Evidence-Based Medicine Domain Knowledge (Series 3): visualization for dissemination of evidence-based medicine information. Chin J Evid-based Med. 2011, 11 (8): 858-867.Google Scholar
- Dong JW: International Statistical Classification of Diseases and Related Health Problems. 2008, Beijing: People’s Health Publishing HouseGoogle Scholar
- Wang J, Liu Q, Wong CG, Wang Y, Li L, Lei X, Zhang F: Quality assessment for Chinese systematic reviews/meta-analyses in public health. Chin J Evid-based Med. 2010, 12: 1367-1374.Google Scholar
- An N, Xu JF, Ge L, Liang L, Shi XT, Zhou WW, Liu YC, Ma JC, Tian JH: Report quality assessment of systematic reviews or meta-analyses of interventions published in Chinese journal of evidence-based pediatrics. Chin J Evid Bases Pediatr. 2013, 8 (2): 87-92.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.