- Open Access
Determining the gaps between Cochrane reviews and trials of effectiveness of interventions for acute respiratory infections: an audit
Systematic Reviews volume 6, Article number: 82 (2017)
Cochrane primarily aims to systematically review trials of effectiveness that are important to inform clinical decisions. Editorial groups support authors to achieve high-quality reviews and prioritise review proposals in their clinical domain that are submitted or elicited. Prioritising proposals requires two approaches, identifying (1) clinical practises for which the evidence of effectiveness is uncertain and (2) interventions in which there are trials of effectiveness (especially randomised controlled trials (RCTs)) not systematically reviewed. This study addresses this second approach for the Cochrane Acute Respiratory Infections Group (CARIG) in order to identify RCTs of acute respiratory infections that have not been systematically reviewed.
We exported, on the 9th of September 2014, and then compared the group’s trials register of RCTs against a list of current Cochrane ARI (systematic) Reviews to identify gaps in topics (the same intervention and health condition) where completed trials have not been systematically reviewed. We assigned a principle intervention and health condition to each of 157 Cochrane reviews (CRs) and 5393 RCTs.
A majority of topics had been systematically reviewed; however, a substantial number (2174 or 41%) of RCTs were not included in any review. The topic that had been systematically reviewed the most was antibiotic vs placebo for pneumonia with 11 CRs and 205 RCTs. The topic that was the subject of most RCTs was vaccination for influenza with 525 RCTs and 6 CRs. Also, 6 CRs had no RCTs (‘empty reviews’).
We identified many RCT topics that have not been systematically reviewed. They need to be addressed in a separate process to establish their priority to clinicians.
Systematic reviews (SRs) summarise and synthesise randomised controlled trials (RCTs), the best method for testing interventions, to produce high levels of evidence. Cochrane is an organisation committed to generating the highest level of evidence by systematically reviewing the medical literature . It comprises 53 editorial groups. The Cochrane Acute Respiratory Infections Group (CARIG) focuses on reviewing and summarising the evidence of treatments for acute respiratory infections (ARIs) . ARIs carry a large burden of disease .
Cochrane reviews are among the most rigorous forms of systematic review  and, accordingly, require the greatest editorial support provided by the Cochrane ARI Group. They typically take 6 months to 2 years to complete [5, 6]. Not all potential titles submitted can be supported, and so, a priority-setting process is necessary. [7–10]. As part of the CARIG’s priority-setting process, we resolved to determine interventions and health conditions for which there are RCTs not systematically reviewed by Cochrane.
Study selection and categorisation
We exported Cochrane reviews specific to the CARIG by interrogating Cochrane’s management software (‘Archie’), equivalent to searching the Cochrane Database of Systematic Reviews (CDSR) Issue 9 of 12, September 2014. We also exported a list of ARI-specific RCTs from the CARIG trials register (the date range of trials was 1930 to 2014). Both exports were done on the 9th of September 2014. Both lists were imported via a reference manager (EndNote) into a spreadsheet where two authors (JA and SV) independently examined the titles (and if necessary, abstracts) to classify the main interventions and health conditions (together forming a ‘topic’). Disagreements were settled by consensus or resolved by a third author (CDM).
We sorted CRs and RCTs by topic (the same intervention and health condition) and then matched CRs and RCTs with the same topic (or paired intervention and health condition). This process enabled us to identify the CRs and RCTS where a match was made on the same topic (intervention and health condition matched) and where there were existing RCTs on particular topics (intervention and health condition) but no CRs. We also identified where there were CRs on particular topics but no RCTs (empty reviews).
Studies that did not consistently cover a single disease or intervention
For each CR or RCT, the intervention was classified followed by the health condition. Where more than one health condition and/or intervention was represented in a single CR or RCT, all health conditions and interventions were classified.
Resolving categorisation discrepancies
Once all studies had been assigned an intervention and health condition, the categories were checked for consistency. Where appropriate, categories were merged together (e.g. the physiotherapy and exercise categories were merged into a single category called ‘physiotherapy/exercise’). Medical and common terms (e.g. ‘pharyngitis’ and ‘sore throat’) were also combined into a single category.
Out of 162 Cochrane reviews screened, 5 were excluded as they were either withdrawn or out of date, leaving 157 for inclusion. Out of 5393 RCT titles screened, 108 were excluded due to not addressing an ARI, had no intervention or were not an RCT, leaving 5285. Of these 409, required reading the abstract and, or, full text.
We initially listed 54 Cochrane review intervention categories, which we merged into 45, and 35 health condition categories were merged into 27. Similarly, 377 RCT intervention categories were merged to 182, and 168 health condition categories to 101.
The most common topics systematically reviewed by the CARIG were antibiotics for pneumonia (11 CRs, 6.4% of the total); vaccination for influenza (n = 6, 3.5%); vaccination for pertussis (n = 4, 2.3%); antiviral drugs for influenza (n = 4, 2.3%); antibiotics for otitis media (n = 4, 2.3%); and antibiotics for sore throat (n = 4, 2.3%) (Table 1). The most common interventions reviewed by the CARIG is antibiotic vs placebo (5 out of the top 10 most common topics). The topics which had been the focus of the most RCTS were vaccinations for influenza (525, 7.7%); vaccination for pertussis (303, 4.4%); and antibiotic vs antibiotic for pneumonia (269, 4%) (Table 2). The most commonly occurring intervention in the ARI trials register was vaccination (6 out of the top 10 most common topics). However, antibiotic vs antibiotic in general was the least common intervention Cochrane reviewed (only 1, 0.6%).
There were many RCTs with no corresponding CRs (2174 or 41%) (Table 3). Most used the intervention of antibiotics, which accounts for 878 RCTs (12.8%). Similarly, there were (only) 6 CRs which reviewed no RCTs (that is they were ‘empty reviews’) (Table 4).
We devised a novel method of representing the extensive relationship between CRs and RCTs, which conveys the information in Tables 1, 2 and 3 that allows the user to interact with the data (Figs. 1, 2 and 3). This is an interactive online graph available from our website .
We found many topics which have been trialled but not reviewed, consistent with the previous findings . These data should allow the CARIG to identify clinical questions in need of review.
The topic with the most Cochrane reviews was antibiotics for pneumonia. Pneumonia makes an important contribution to the burden of disease worldwide, especially in the developing world , and so, this over-representation seems appropriate. The interventions trialled with least representation with CRs, antibiotic vs antibiotic, are often driven by pharmaceutical companies (interested in demonstrating that a new member of an existing antibiotic class has equivalent efficacy), something perhaps less interesting to clinicians.
Strengths of our methods were the exploitation of the set of trials already collected by Cochrane and the collection of CRs and our matching methods and online visualisation techniques. Weaknesses include the limitation to CRs (there are undoubtedly other systematic reviews outside Cochrane), the potentially arbitrary over-simplification of the topics to one or two interventions for each health condition category and our limitation to treatment questions (Cochrane has a minority of diagnostic reviews as well)—nor did we account for ‘stabilised reviews’ (those in which the intervention is no longer current, e.g. amantadine and rimantadine for influenza  or where there is sufficient evidence to settle for a clinical question), e.g. vaccination to prevent polio .
These data will inform our forthcoming priority-setting exercise during which they will be presented to stakeholders (health consumers and clinicians) to allow judgement to be made about which topics should be given higher priority.
Acute respiratory infections
Cochrane Acute Respiratory Infections Group
Cochrane Database of Systematic Reviews
Randomised controlled trial
Cochrane: about us. http://www.cochrane.org/about-us. Accessed 22 Dec 2016.
Cochrane Acute Respiratory Infections: about us. http://ari.cochrane.org/welcome. Accessed 22 Dec 2016.
Ferkol T, Schraufnagel D. The global burden of respiratory disease. Ann Am Thorac Soc. 2014;11(3):404–6.
Useem J, Brennan A, LaValley M, Vickery M, Ameli O, Reinen N, Gill CJ. Systematic differences between Cochrane and non-Cochrane meta-analyses on the same topic: a matched pair analysis. PLoS One. 2015;10(12):e0144980.
Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci. 2010;5:56.
Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, Perrier L, Hutton B, Moher D, Straus SE. A scoping review of rapid review methods. BMC Med. 2015;13:224.
Cochrane Methods priority setting: background & history. http://methods.cochrane.org/prioritysetting/background-history. Accessed 18 Dec 2016.
Synnot A. Stakeholder priorities for research in health communication and participation: findings from the Cochrane consumers and communication priority setting project. Melbourne: Centre for Health Communication and Participation, La Trobe University; 2016.
Cochrane tobacco addiction: priority setting workshop http://tobacco.cochrane.org/priority-setting-workshop. Accessed 16 Dec 2016.
Welsh E, Stovold E, Karner C, Cates C. Cochrane Airways Group reviews were prioritized for updating using a pragmatic approach. J Clin Epidemiol. 2015;68(3):341–6.
Cochrane ARI Group. Topics covered (systematic reviews) vs. not-covered (no systematic reviews) by the Cochrane ARI Group http://crebp.github.io/CREBP-Disease-Treatment/. Accessed 15 Jan 2017.
Glasziou P, Haynes B. The paths from research to improved health outcomes. ACP J Club. 2005;142(2):A8–10.
Jefferson T, Demicheli V, Di Pietrantonj C, Rivetti D. Amantadine and rimantadine for influenza A in adults. Cochrane Database Syst Rev. 2006 19;(2):CD001169.
CDC. Polio vaccination. https://www.cdc.gov/vaccines/vpd/polio/index.html. Accessed 19 Dec 2016.
Alloo J, Vallath S, Del Mar C, Carter M, Thorning S, Clark J. Determining the gaps between Cochrane reviews and trials of effectiveness of interventions for acute respiratory infections: an audit (Data set). In: Gold Coast: Bond University; 2016.
The authors would like to thank Dr Anna Mae Scott from The Centre for Research Excellence in Minimising Antibiotic Resistance from Acute Respiratory Infections (CREMARA) for providing valuable input on the draft of the manuscript.
At the time of the study, JA and SV were medical students and not funded. CDM, ST, MC and JC were funded by the National Health and Medical Research Council (NHMRC).
Availability of data and materials
CDM, ST, JA, SV and MC contributed to the conception and design of the work. JA, SV and JC conducted the data analysis. MC devised the graphical analysis of the data. All authors drafted the work or revised it critically for important intellectual content, gave the final approval for the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated or resolved.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Alloo, J., Vallath, S., Del Mar, C. et al. Determining the gaps between Cochrane reviews and trials of effectiveness of interventions for acute respiratory infections: an audit. Syst Rev 6, 82 (2017). https://doi.org/10.1186/s13643-017-0472-0
- Cochrane reviews
- Research gaps
- Research prioritisation