Skip to main content

The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research

Abstract

Background

There has been a growing emphasis on evidence-informed decision-making in health care. Systematic reviews, such as those produced by the Cochrane Collaboration, have been a key component of this movement. The UK National Institute for Health Research (NIHR) Systematic Review Programme currently supports 20 Cochrane Review Groups (CRGs). The aim of this study was to identify the impacts of Cochrane reviews published by NIHR-funded CRGs during the years 2007–2011.

Methods

We sent questionnaires to CRGs and review authors, interviewed guideline developers and used bibliometrics and documentary review to get an overview of CRG impact and to evaluate the impact of a sample of 60 Cochrane reviews. We used a framework with four categories (knowledge production, research targeting, informing policy development and impact on practice/services).

Results

A total of 1,502 new and updated reviews were produced by the 20 NIHR-funded CRGs between 2007 and 2011. The clearest impacts were on policy with a total of 483 systematic reviews cited in 247 sets of guidance: 62 were international, 175 national (87 from the UK) and 10 local. Review authors and CRGs provided some examples of impact on practice or services, for example, safer use of medication, the identification of new effective drugs or treatments and potential economic benefits through the reduction in the use of unproven or unnecessary procedures. However, such impacts are difficult to objectively document, and the majority of reviewers were unsure if their review had produced specific impacts. Qualitative data suggested that Cochrane reviews often play an instrumental role in informing guidance, although a poor fit with guideline scope or methods, reviews being out of date and a lack of communication between CRGs and guideline developers were barriers to their use.

Conclusions

Health and economic impacts of research are generally difficult to measure. We found that to be the case with this evaluation. Impacts on knowledge production and clinical guidance were easier to identify and substantiate than those on clinical practice. Questions remain about how we define and measure impact, and more work is needed to develop suitable methods for impact analysis.

Peer Review reports

Background

In recent years, there has been a growing emphasis on the use of evidence to inform decision-making in health care[1–3], with the use of evidence seen as particularly relevant to commissioning because of the large financial commitments involved and because of the increasing complexity of health-care management decisions[4]. In addition, there has been growing interest in the way in which research is used, with researchers increasingly expected to consider the wider impacts of their work[5]. This may include the contributions research makes to health, society, culture, the economy, quality of life and public policy.

One of the key aspects of evidence-informed policy and practice has been the development of methods for the synthesis and integration of primary research, in the form of systematic reviews. Systematic reviews have been regarded as particularly important tools for decision-makers as it inherently makes sense for decisions to be based on the totality of evidence rather than a single study[6, 7]. However, despite this, the extent to which policy makers and practitioners use systematic reviews as a source of evidence has been questioned[4, 6]. Indeed, it has long been recognised that research may not always have the impact that researchers desire[8, 9].

The Cochrane Collaboration is an independent, international organisation involved in preparing, maintaining and disseminating systematic reviews evaluating the effectiveness of health-care interventions. Cochrane systematic reviews should be uniquely placed to influence policy, practice and research as they provide a comprehensive, critical summary of what is known about effectiveness on a given topic; the rigour of their methods is widely acknowledged, and they are periodically updated in light of new evidence. However, whilst it is acknowledged that Cochrane groups produce high-quality systematic reviews[10–12], there is at present a lack of information about the impacts of Cochrane reviews.

The UK National Institute for Health Research (NIHR) systematic review programme currently supports 20 Cochrane Review Groups (CRGs) that have their editorial bases in academic or health institutions in the UK. These groups cover a broad range of health-care areas and produce almost half of all Cochrane reviews. This study was commissioned by the NIHR to identify the impacts and likely impacts of Cochrane reviews published by the 20 NIHR-funded CRGs between the years 2007 and 2011. The aims of this study were to identify impacts on clinical guidance and clinical practice and identify important gaps in knowledge and possible influence on the conduct of new primary research studies. In addition, we sought to identify barriers and facilitators to Cochrane reviews being used by guideline developers.

Methods

We used a mixed methods approach informed by theories about research impact[9, 13–15] and guided by a framework that draws on previous work around the evaluation of research impact[16–19]. This framework, including main and subcategories, can be seen in Table 1. A variety of methods exist for evaluating research impact including bibliometrics, documentary analysis, semi-structured interviews, case studies and surveys[20, 21]. As there are advantages and disadvantages of each method, it is generally recommended that a variety of sources to identify research impact are used[22, 23]. We used a mixture of bibliometrics, documentary analysis, questionnaire surveys and interviews. Obtaining the ‘insider account’ has been recommended when evaluating research impact[21], and it was envisaged that staff at editorial bases and review authors would be important sources of information about review impact.There were two main components to the study. In the first (phase 1), we aimed to obtain a general overview of the impact of the outputs produced by NIHR-funded CRGs, and in the second (phase 2), we undertook a more detailed evaluation of a sample of Cochrane reviews. The study was conducted between April and September 2013 and only included reviews first published or updated during the years 2007–2011 (time period was stipulated by the funder). The evaluation covered the following CRGs: Airways; Bone, Joint and Muscle Trauma; Cystic Fibrosis and Genetic Disorders Group; Dementia and Cognitive Improvement; Depression, Anxiety and Neurosis; Ear, Nose and Throat; Epilepsy; Eyes and Vision; Gynaecological Cancer; Heart; Incontinence; Injuries; Neuromuscular; Oral Health; Pain, Palliative and Supportive Care; Pregnancy and Childbirth; Schizophrenia; Skin; Tobacco Addiction; and Wounds. For an overview of study methods, including how each part informs the findings, see Figure 1.

Table 1 Evaluation framework
Figure 1
figure 1

Overview of methods.

Ethical approval for the study was obtained from the University of Hertfordshire Health and Human Sciences Ethics Committee with delegated authority (ECDA), reference number HSK/SF/UH/00003.

Methods for phase 1: overview of impact of CRG outputs (2007–2011)

Questionnaire survey

We sent a questionnaire survey (Additional file1) to the editorial base (n = 20) of each NIHR-funded CRG. The questionnaires were sent via email with a personalised covering letter explaining the purpose of the study. The survey included questions about general impact (both actual and potential) of the CRG output between the years 2007 and 2011; the questions were informed by our evaluation framework. Respondents were asked, where possible, to provide supporting evidence of impact. This could include citation in clinical guidelines, impact on practice (for example, changes to clinicians behaviour, changes to service organisation and delivery), or influence on future primary research. CRGs were also asked to identify reviews published (or updated) in the last 5 years that they considered to have had the most impact on policy and practice.

Documentary review and analysis of existing sources

In addition to the questionnaires, we hand-searched the annual reports that CRGs had provided to the NIHR Health Technology Assessment (HTA) programme for the years 2007–2012, reviewed data on the use of Cochrane reviews in National Institute for Health and Care Excellence (NICE) and Scottish Intercollegiate Guidelines Network (SIGN) guidance compiled by the UK Cochrane Centre and searched the Cochrane Quality, Innovation, Productivity and Prevention (QIPP) topics database via NHS Evidence (https://www.evidence.nhs.uk/qipp). This documentary analysis was supplemented with keyword searches on a number of databases and internet sites including Google, NHS Evidence, and the World Health Organisation (WHO).

Data verification and analysis

Data from the questionnaires were entered on an Excel spreadsheet. One researcher extracted examples of actual or potential impact for each CRG and recorded this on a specially designed data extraction form based on the categories from our framework (Additional file2). Additional data identified via documentary review was added to this form. Data relating to guidelines were collated in a separate Excel spreadsheet which was stratified by CRG. Data, and how to interpret it, was discussed with a second member of the team. Information was critically assessed and, where possible, evidence sought to verify it. For example, when a review was said to have led to follow-on research, we searched for the study protocol or final report in order to check that the review had been cited as justification for the research, and we obtained hard copies of guidelines and searched using the word ‘Cochrane’ to verify which reviews had been cited and whether they were within the time frame for this evaluation (2007–2011). We excluded examples where no supporting evidence was available.

Qualitative interviews

We undertook telephone interviews with a convenience sample of guideline developers; this included those involved in developing and managing guidelines and systematic reviewers and technical advisors producing evidence reviews to underpin guidance. In the UK, participants were recruited from NICE (or one of their collaborating centres) or SIGN, and international participants were recruited via the WHO.

Potential participants were identified by one of the authors with further snowballing as required. Our approach was iterative, and recruitment was stopped when we felt that we had reached data saturation. We used a semi-structured interview schedule which included questions about how systematic reviews were used in the development of guidance, experiences of using Cochrane reviews and ways in which use of Cochrane reviews might be facilitated. Interviews lasted between 20 and 40 min and were taped and transcribed. Informed consent was obtained from each participant.

Analysis

We drew on thematic content analysis to enable key features of guideline developers’ experiences of using Cochrane reviews to be elicited from the data[24]. To guarantee a degree of inter-rater reliability and transparency, two researchers independently scrutinised each transcript and applied open codes to text. From this, a list of initial codes and themes were created. This was refined after further discussion with the wider project team, and supporting evidence in the form of quotes was documented.

Methods for phase 2: evaluation of impact of representative sample of Cochrane reviews (2007–2011)

In phase 2, we undertook further analysis on a purposive sample of 60 Cochrane reviews published or updated between 2007 and 2011. For each CRG, we chose one review randomly and two on the basis that they had potentially had an impact. Where possible, we used the information provided by CRGs in their questionnaires to inform the selection of reviews. However, in some cases, CRGs did not provide this information and we used other sources, such as citation counts or data from Wiley (publishers of the Cochrane Library), to guide our selection. As impact may take some time to build, we weighted our sample towards those reviews published between the years 2007 and 2010.

Questionnaire survey with systematic review authors

We sent a questionnaire survey (Additional file3) to first authors of all 60 reviews. Questions were similar to those in the CRG questionnaire with authors asked to consider impact on knowledge production and possible impact of the review on health policy and practice. Authors were also asked if they thought their review identified important gaps in knowledge and/or if it had any influence on the conduct of new primary research. Methods for analysis and data verification were similar to those for the CRG questionnaire. Non-responders received a second mailing from the researchers and, in addition, 26 authors were followed up by the Managing Editors at CRG editorial bases.

Documentary and bibliometric analysis

We undertook citation analysis in Web of Science (WoS), Scopus and Google Scholar (searches conducted May/June 2013); undertook searches on Google, NHS Evidence and TRIP (http://www.tripdatabase.com/) using review author and title keywords (between May and July 2013); and reviewed data from the publishers of The Cochrane Library (Wiley) on review downloads and media mentions. We also used an alternative metric measure (http://www.altmetric.com/). This provides article level metrics which give an indication of the impact of a publication by looking at activity surrounding the publication on social media sites and in policy documents and newspapers. Articles for which no mentions have been recorded score 0. Results from the documentary and bibliometric analyses were entered into a specially designed data extraction form in Access (see Additional file4). Data on citation counts was entered by one researcher and checked by a second.

Results

A total of 3,187 new and updated reviews were published on the Cochrane Database of Systematic Reviews between 2007 and 2011, 1,502 (47%) of which were produced by the 20 CRGs funded by the NIHR (see Figure 2).

Figure 2
figure 2

New and updated reviews published between 2007 and 2011 (stratified by CRG).

Seventeen of the 20 (85%) CRGs and 29 (48%) of review authors returned questionnaires. There was some variation in the number of author questionnaires returned for each CRG with all three questionnaires returned for some groups and none for others. Of the 60 reviews selected for analysis, 9 (15%) were updates and the rest were new reviews. Thirteen (22%) were published in 2007, 23 (38%) in 2008, 12 (20%) in 2009 and 6 (10%) in 2010 and 2011, respectively. Thirty-four (57%) of the first authors were based in the UK. Details of these reviews, including title, country of first author and whether they were a new review or an updated review, can be seen in Additional file5.

Knowledge production, identifying gaps in the evidence and stimulating research

CRG and author questionnaires provided 40 examples where they felt reviews had influenced primary research, and 13 (22%) of the sample of 60 reviews had been cited in a protocol or the background of a primary research study. In general, most of these examples related to work conducted by the Cochrane reviewers themselves; respondents to the questionnaires were less sure if their review had influenced the research of others. Most of the examples of follow-on research were RCTs. A summary by CRG can be seen in Table 2.Although there was considerable variation between the reviews, the data do suggest that many of the 60 reviews have been of interest to decision makers. For example, 27 (45%) of the 60 reviews had had 100 or more citations in Google Scholar and 5 had received over 400 citations. Citation counts were higher in Google Scholar than in WoS or Scopus. The number of downloads from the Cochrane Library varied considerably between reviews. Of the sample of 60 reviews, the ten that were downloaded most frequently (full text and abstract) for the years 2009–2011 can be seen in Figure 3. This figure give an indication of the impact of reviews within the research and practice communities and show how downloads for reviews have increased over the 5-year period.

Table 2 Number of reviews informing primary research
Figure 3
figure 3

Top ten downloads from 2009 to 2011.

Thirty-six (60%) of the 60 reviews had an Altmetric score of 1 or more with 12 (20%) having a score over 10, and 4 (7%) scoring over 50. Currently, mid-tier publications might expect 30%–40% of papers to have a score of at least 1[25]. Although these scores need to be interpreted cautiously, for example, they may not accurately reflect the interest in reviews published before 2011, they do provide some indication of the interest around a review and the way Cochrane reviews may have impacted on knowledge production by stimulating discussion and debate[26]. Citations counts and Altmetric scores for each review can be seen in Table 3 with a summary of citation analysis data in Table 4.

Table 3 Number of citations (WoS, Scopus and Google Scholar) and Altmetric score
Table 4 Summary of citation analysis data from WoS, Scopus and Google Scholar

Informing policy development

Systematic reviews from all the 20 CRGs were cited in some form of clinical or practice guidance. Across the CRGs, there were 722 citations in 248 guidelines (or in the evidence reviews used to develop the guidance) with 481 systematic reviews being cited at least once. Of these, 62 were international guidance, 175 national guidance, and 10 local guidance (e.g. at Trust or hospital level). Of the national guidance, 87 were developed in the UK, with Cochrane reviews cited in 30 sets of NICE guidance and 23 sets of SIGN guidance. Information on inclusion in guidelines (by CRG) is summarised in Table 5.

Table 5 Summary of information relating to inclusion of reviews in guidelines (stratified by CRG)

We interviewed eight participants, four from NICE (or NICE collaborating centres), two from SIGN and two from the WHO. Five of the interviewees were female and three were male. Two participants were senior managers involved at a strategic level in co-ordinating guideline development, and six were involved in the production of individual guidelines either as managers, systematic reviewers or technical advisors. More details of the participants and their roles can be seen in Table 6. Analysis of the interview transcripts resulted in six overarching themes and a number of subthemes relating to the views and experiences of guideline developers and their use of Cochrane reviews. The overarching themes are the following: the process of using Cochrane reviews in the development of guidance, the quality of Cochrane Reviews, culture and approaches, up-to-date evidence, methodological issues and collaboration and communication. These themes and subthemes can be seen in Table 7. Whilst a number of the issues that arose might be applicable to systematic reviews in general, the focus of our questions and analysis was on the use of Cochrane reviews.

Table 6 Details of interview participants
Table 7 Results of thematic analysis and barriers and facilitators to the use of Cochrane reviews in the development of guidance

Results from the semi-structured interviews suggest that searching for relevant Cochrane reviews is part of the guideline development process and that Cochrane reviews often play an instrumental role in informing guidance. Cochrane reviews appeared to be used at a number of different stages of the guideline development process, for example, being used early in the process to scope review questions and assess the strength of the evidence and later in the process as part of the evidence review to develop the guidance. Even when the whole Cochrane review was not used, guideline developers often drew on component parts of the review such as search strategies, lists of included and excluded studies, quality assessment data and analyses. However, there were a number of barriers to the use of Cochrane reviews in guidance. Cochrane reviews might not be available, they might not fit with the guideline scope, they may be out of date, or the methods used may not fit with those required for the guideline (see Table 7). Evidence to support the thematic analysis can be seen in Additional file6.

Impact on clinical practice and services

There was evidence to suggest that some Cochrane reviews may have contributed to a number of benefits to the health service including safer or more appropriate use of medication or other health technologies or the identification of new effective drugs or treatments (see the list below). Eight CRGs and 12 authors gave examples of impact on practice or services. However, it was difficult to verify some of these impacts and to judge whether these changes were directly attributable to the Cochrane review(s). Eighty-three percent of review authors who responded to the questionnaire were unsure if their work had changed the behaviour of practitioners, managers or members of the public, or if their work had helped reduce costs (69%), increase quality (76%), improve effectiveness (76%) or promote equity (79%).

  •  A review on support surfaces for pressure ulcer prevention[27] was used to inform guidance on purchasing within the NHS[28].

  •  Reviews on long-acting beta-antagonists (LABA) in asthma[29–31] may have led to safer prescribing of these drugs for people with asthma (http://www.fda.gov/Drugs/DrugSafety/PostmarketDrugSafetyInformationforPatientsandProviders/ucm200776.htm).

  •  A review on colloids versus crystalloids for fluid resuscitation[32, 33] may have influenced calls to stop starch use within the NHS, a decision that has the potential to save both lives and money[34].

  •  An updated review on antiviral treatment for Bell’s palsy[35] may have contributed to changes in practice and a reduction of prescriptions of antiviral drugs for Bells palsy (http://cks.nice.org.uk/bells-palsy).

  •  A review on antifibrinolytic drugs for trauma patients[36, 37] led to follow-on research which influenced the decision by the Medicines Innovation Scheme to fast track tranexamic acid for use in the NHS. Ambulance crews throughout the NHS now administer tranexamic acid to bleeding trauma patients (http://www.swast.nhs.uk/txa.htm).

Potential benefits of Cochrane reviews were highlighted in the NICE Cochrane Quality and productivity topics, 19 of which related to reviews produced by one of the 20 CRGs during the years 2007–2011. Potential benefits identified included economic benefits through budget savings or the release of funds, improvements in clinical quality, the reduction in the use of unproven or unnecessary procedures and improvements in patient and carer experiences.

Discussion

Summary of findings

There was evidence that reviews from all the CRGs had had some impact. The clearest impacts were on health-care policy, with less evidence of direct impact on research targeting, clinical practice and the organisation and delivery of NHS services. From our sample of 60 reviews, there was evidence to suggest that some reviews have had a significant impact on the research and practice communities, whilst others appear to have had little or no impact (a summary of the main impacts of the 60 reviews can be seen in Additional file7). Some of the reviews associated with the clearest evidence of impact had been updated during 2007–2011 rather than first published during that time[33, 38, 39].

There was considerable variation in evidence of impact between CRGs and between reviews. Variation between CRGs might be accounted for by differences in the scope or speciality of the group, the type and number of outputs, or methods used for dissemination and knowledge transfer. However, it may also be a reflection of whether CRGs routinely collect data on impact. Variation amongst the sample of 60 reviews might be a reflection of the relevance of the review findings to decision makers, the date of publication, the strength of the evidence or the strategies used to disseminate the findings.

Knowledge production and research targeting

We found evidence to suggest that some Cochrane reviews had played a role in identifying gaps in the evidence and stimulating new research. Across the CRGs, there were 40 examples of reviews influencing primary research. However, most of these examples of research impact related to work conducted by the Cochrane reviewers themselves; there was less evidence of a broad impact on stimulating new research.

Based on the assumption that influential or important work will be cited more frequently than others[40] and that alternative metrics give an indication that papers have been read and discussed[26], the results of the bibliometric analysis suggest that a number of the reviews had had an impact on the creation of new knowledge and the stimulation of discussion and debate. Citation counts were much higher in Google Scholar than in WoS or Scopus. Google Scholar may be of particular importance to citation analyses for Cochrane reviews as previous work[41, 42] suggests that citation counts for Cochrane reviews are artificially low in ISI databases and Scopus because citing authors have incorrectly referenced Cochrane reviews. However, as there are also some concerns about the accuracy of Google Scholar[43], evaluations of citation data for Cochrane reviews should include more than one database.

Informing policy development

Reviews from all the CRGs were cited in some form of clinical or practice guidance. Mostly, this was national and international guidance, with only ten examples of local guidance. This may be because local guidance is often not available outside of the organisations involved and so may be more difficult to find. CRGs and review authors gave us a number of anecdotal examples of reviews influencing local guidance (for example, at hospital or department levels), but most were excluded from our final numbers as we were unable to find a copy of the guidance in order to verify that it referenced a Cochrane review. It is possible, therefore, that the impact of Cochrane reviews on local guidance is underestimated in our evaluation.

The fact that a review is cited in a guideline does not mean that it was instrumental in the development of the final guidance. Conversely, a Cochrane review might have been used to inform policy development but may not be referenced or cited. Interviews with guideline developers suggest that Cochrane reviews are routinely used to inform the guideline development process. However, reviews with a narrow focus or that are out of date are of less use to decision maker.

Impact on clinical practice and services

Review authors who responded to the questionnaire were generally unsure if their work had changed the behaviour of practitioners, managers or members of the public, or if their work had helped reduce costs, increase quality, improve effectiveness or promote equity. CRGs and authors did provide some examples to suggest that Cochrane reviews had contributed to a number of benefits to the health service including safer or more appropriate use of medication or other health technologies or the identification of new effective drugs or treatments, or that they had the potential to lead to cost-saving and health-service benefits. However, attributing particular behaviour changes, health benefits or cost saving to a particular systematic review (or reviews) is difficult. Generally, new research adds to an existing pool of knowledge[44] and many research projects may lie behind a specific advance in health care[45]. Many Cochrane reviews have a narrow focus, often deliberately. Whilst, this may ensure internal validity, it may reduce its impact in the policy and practice arenas, a factor recognised in the move by the Cochrane Collaboration to introduce overviews of reviews[46].

Strengths and limitations

Evaluating the impact of research is complicated. At present, there are no agreed instruments or methods for determining impact[47], and it is acknowledged that knowledge production is more easily discernible than impact on policy or health gain[21]. Moreover, although health benefits and broader economic benefits may be viewed as the real ‘payback’ from health research, these are hard to measure as it is difficult to attribute particular health gains to specific pieces of research[23]. Although we were able to make some inferences about health and economic benefits, these were largely beyond the remit of this study.

Our approach included the use of citation analysis and documentary review. Bibliometric techniques have been criticised for focusing on quantity rather than quality, and measuring the number of research outputs rather than research outcomes or impact[20]. Moreover, it may take several months or even years for a work to be first cited. However, we combined traditional citation counts with the use of alternative metrics. This gave us another measure of the amount of attention a review had received including its impact in social media. In addition, our mixed methods approach, including documentary review, questionnaires with CRGs and review authors, and interviews with guideline developers, enabled us to get a more complete picture of impact.

We interviewed only eight guideline developers, six of whom were based in the UK, and all of whom came from leading guideline development agencies. It is possible that their views and experiences may not be directly transferable to other organisations or to guideline developers in other countries. However, we found consistent and recurring themes in the interviews, and the views expressed are concordant with literature in this area[48, 49].

We examined the impact of three reviews per CRG. Our sample is relatively small, and it is possible that the reviews we chose are not typical of the outputs of those CRGs, that there are better examples of impact and that the findings are not generalizable. Moreover, only 48% of the authors responded to the survey which raises further questions about the generalizability of the findings. However, this response rate compares favourably with previous surveys of Cochrane authors[50].

Although most of the data in the report are presented stratified by CRG, direct comparisons between reviews or between CRGs may not be appropriate. The diverse nature of the review topics and the differences in the number of questionnaires returned for each group and in length of time since publication mean that one should be cautious when comparing the impact of different reviews. Citation volume typically peaks in the third or fourth year post-publication, and therefore, a window of 5 years has been suggested as most appropriate for research assessment[51, 52]. Reviews published more recently may not yet have had time to impact on policy or practice. Indeed, it has been suggested that it may take up to a decade for the full impact of research to be apparent[53].

There are several potential sources of bias in this evaluation. This was a retrospective analysis and as such may be at greater risk of bias than one where data is collected prospectively. In addition, much of our data was obtained from questionnaires to CRGs and review authors, and there is a risk of recall bias or that respondents might have inflated the impact of their work. In order to overcome this, the research team critically assessed all information provided and sought evidence to verify impacts and check whether it was related to an output produced in the time frame of interest.

Implications of the findings

Our study provides evidence that Cochrane reviews produced by NIHR-funded CRGs have an impact on clinical guidance development and may have influenced the conduct of primary research. We found limited evidence that Cochrane reviews had had a direct effect on clinical practice, but they may have an indirect impact on health care and patient outcomes through their role in informing further primary research and clinical guidance. Although the implementation of NICE guidance has been shown to be variable[54, 55], there is evidence that clinical guidelines can be effective in changing the process and outcome of care[56–60].

There are significant difficulties associated with determining the impact of specific pieces of research[21]. The focus of this evaluation was on the outputs of 20 CRGs covering a broad range of health-care topics. It is possible that it would be easier to determine impact on clinical practice and the behaviour of health-care providers if such evaluations had a narrower focus. This might allow the use of more qualitative and quantitative methods targeted at specific groups of health-care providers.

Interviews with guideline developers suggest that Cochrane reviews are routinely used in the guideline development process. However, reviews with a narrow focus or that are out of date are of less use to decision makers. Use of Cochrane reviews in the development of guidelines might be facilitated by better collaboration and communication between CRGs and guideline developers. Formal as well as informal networks might be needed to facilitate the transfer of research knowledge to decision makers[61].

Conclusions

This study identified a number of impacts and likely impacts of Cochrane reviews. The clearest impacts of Cochrane reviews were on health-care policy, with less evidence of a direct impact on clinical practice and the organisation and delivery of NHS services. Whilst it is important for researchers to consider how they might increase the influence of their work, such impacts are difficult to measure. Questions remain about how we define and measure impact, and more work is needed to develop suitable methods for evaluating the impact of systematic reviews.

References

  1. Ham C, Hunter DJ, Robinson R: Evidence based policymaking. BMJ. 1995, 310 (6972): 71-72. 10.1136/bmj.310.6972.71.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A, Osborne J: Sicily statement on evidence-based practice. BMC Med Educ. 2005, 5 (1): 1-10.1186/1472-6920-5-1.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn’t. BMJ. 1996, 312 (7023): 71-72. 10.1136/bmj.312.7023.71.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Swan J, Clarke A, Nicolini D, Powell J, Scarborough H, Roginski C, Gkeredakis E, Mills P, Taylor-Phillips S: Evidence in Management Decisions (EMD)—Advancing Knowledge Utilization in Healthcare Management. Final Report. NIHR Health Service and Delivery Research Programme 2012, [http://www.netscc.ac.uk/hsdr/files/project/SDO_FR_08-1808-244_V01.pdf],

  5. HEFC: Research Excellence Framework: Second Consultation on the Assessment and Funding of Research. 2009, London: Higher Education Funding Council for England

    Google Scholar 

  6. Sheldon TA: Making evidence synthesis more useful for management and policy-making. J Health Serv Res Policy. 2005, 10 (Suppl 1): 1-5. 10.1258/1355819054308521.

    Article  PubMed  Google Scholar 

  7. Black N: Evidence based policy: proceed with care. BMJ. 2001, 323 (7307): 275-279. 10.1136/bmj.323.7307.275.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Lomas J: Using ‘linkage and exchange’ to move research into policy at a Canadian foundation. Health Aff (Millwood). 2000, 19 (3): 236-240. 10.1377/hlthaff.19.3.236.

    Article  CAS  Google Scholar 

  9. Weiss C: Using research in the policy process: potential and constraints. Pol Stud J. 1976, 4: 224-228. 10.1111/j.1541-0072.1976.tb01557.x.

    Article  Google Scholar 

  10. Olsen O, Middleton P, Ezzo J, Gotzsche PC, Hadhazy V, Herxheimer A, Kleijnen J, McIntosh H: Quality of Cochrane reviews: assessment of sample from 1998. BMJ. 2001, 323 (7317): 829-832. 10.1136/bmj.323.7317.829.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Moseley AM, Elkins MR, Herbert RD, Maher CG, Sherrington C: Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol. 2009, 62 (10): 1021-1030. 10.1016/j.jclinepi.2008.09.018.

    Article  PubMed  Google Scholar 

  12. Jorgensen A, Hilden J, Gotzsche P: Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ. 2006, 333 (7572): 782-10.1136/bmj.38973.444699.0B.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Huberman M: Linking the practitioner and researcher communities for school improvement. Sch Eff Sch Improv. 1993, 4: 1-16. 10.1080/0924345930040101.

    Article  Google Scholar 

  14. Nutley S, Percy-Smith J, Solesbury W: Models of Research Impact: A Cross-Sector Review of Literature and Practice. 2003, London: Learning and Skills Research Centre

    Google Scholar 

  15. Estabrooks CE: The conceptual structure of research utilization. Res Nurs Health. 1999, 22: 203-216. 10.1002/(SICI)1098-240X(199906)22:3<203::AID-NUR3>3.0.CO;2-9.

    Article  CAS  PubMed  Google Scholar 

  16. Buxton M, Hanney S: How can payback from health services research be assessed?. J Health Serv Res Policy. 1996, 1 (1): 35-43.

    CAS  PubMed  Google Scholar 

  17. Hanney S, Packwood T, Buxton M: Evaluating the benefits from health research and development centres: a categorization, a model and examples of application. Evaluation: Int J Theory Res Pract. 2000, 6 (2): 137-160.

    Article  Google Scholar 

  18. Kuruvilla S, Mays N, Pleasant A, Walt G: Describing the impact of health research: a Research Impact Framework. BMC Health Serv Res. 2006, 6: 134-10.1186/1472-6963-6-134.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Kuruvilla S, Mays N, Walt G: Describing the impact of health services and policy research. J Health Serv Res Policy. 2007, 12 (Suppl 1): S1-10.1258/135581907780318400. -23-31

    Article  PubMed  Google Scholar 

  20. Boaz A, Fitzpatrick S, Shaw B: Assessing the Impact of Research on Policy: A Review of the Literature for a Project on Bridging Research and Policy Through Outcome Evaluation. 2008, London: University of London

    Google Scholar 

  21. Hanney SR, Grant J, Wooding S, Buxton MJ: An assessment of the impact of NHS Health Technology Assessment Programme. Health Technol Assess. 2007, 11 (53): 1-180.

    Article  Google Scholar 

  22. Lavis J, Ross S, McLeod C, Gildiner A: Measuring the impact of health research. J Health Serv Res Policy. 2003, 8 (3): 165-170. 10.1258/135581903322029520.

    Article  PubMed  Google Scholar 

  23. Hanney S, Grant J, Wooding S, Buxton M: Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK’s ‘Arthritis Research Campaign’. Health Res Pol Syst. 2004, 2 (1): 4-10.1186/1478-4505-2-4.

    Article  Google Scholar 

  24. Coffey A, Atkinson P: Making Sense of Qualitative Data: Complementary Research Strategies. 1996, New York: Sage

    Google Scholar 

  25. Altmetric: What does Altmetric do?. [http://www.altmetric.com/whatwedo.php#score],

  26. Piwowar H: Altmetrics: what, why and where. Bull Assoc Inform Sci Technol. 2013, 39 (4): 8-9. 10.1002/bult.2013.1720390404. [https://asis.org/Bulletin/Apr-13/AprMay13_Piwowarhtml],

    Article  Google Scholar 

  27. McInnes E, Bell-Syer SE, Dumville JC, Legood R, Cullum NA: Support surfaces for pressure ulcer prevention. Cochrane Database Syst Rev. 2008, 4: CD001735-

    PubMed  Google Scholar 

  28. Burnett J, Clift M, Sprange K: Buyers Guide: Pressure-Redistribution Mattresses and Overlays CEP08042. 2008, London: NHS Purchasing and Supply Agency, Centre for Evidence-Based Purchasing

    Google Scholar 

  29. Cates CJ, Lasserson TJ: Regular treatment with formoterol and an inhaled corticosteroid versus regular treatment with salmeterol and an inhaled corticosteroid for chronic asthma: serious adverse events. Cochrane Database Syst Rev. 2009, 1: CD007694-

    Google Scholar 

  30. Cates CJ, Lasserson TJ, Jaeschke R: Regular treatment with salmeterol and inhaled steroids for chronic asthma: serious adverse events. Cochrane Database Syst Rev. 2009, 3: CD006922-

    PubMed  Google Scholar 

  31. Cates CJ, Cates MJ: Regular treatment with salmeterol for chronic asthma: serious adverse events. Cochrane Database Syst Rev. 2008, 3: CD006363-

    PubMed  Google Scholar 

  32. Perel P, Roberts I: Colloids versus crystalloids for fluid resuscitation in critically ill patients. Cochrane Database Syst Rev. 2011, 3: CD000567-

    PubMed  Google Scholar 

  33. Perel P, Roberts I: Colloids versus crystalloids for fluid resuscitation in critically ill patients. Cochrane Database Syst Rev. 2007, 4: CD000567-

    PubMed  Google Scholar 

  34. European Medicines Agency: PRAC Recommends Suspending Marketing Authorisations for Infusion Solutions Containing Hydroxyethyl Starch. 2013, London: EMA, [http://www.ema.europa.eu/docs/en_GB/document_library/Referrals_document/Solutions_for_infusion_containing_hydroxyethyl_starch/Recommendation_provided_by_Pharmacovigilance_Risk_Assessment_Committee/WC500144448.pdf],

    Google Scholar 

  35. Lockhart P, Daly F, Pitkethly M, Comerford N, Sullivan F: Antiviral treatment for Bell’s palsy (idiopathic facial paralysis). Cochrane Database Syst Rev. 2009, 4: CD001869-

    PubMed  Google Scholar 

  36. Roberts I, Shakur H, Ker K, Coats T: Antifibrinolytic drugs for acute traumatic injury. Cochrane Database Syst Rev. 2011, 1: CD004896-

    PubMed  Google Scholar 

  37. Roberts I, Shakur H, Ker K, Coats T: Antifibrinolytic drugs for acute traumatic injury. Cochrane Database Syst Rev. 2009, 12: CD004896-

    Google Scholar 

  38. Gillespie LD, Robertson MC, Gillespie WJ, Lamb SE, Gates S, Cumming RG, Rowe BH: Interventions for preventing falls in older people living in the community. Cochrane Database Syst Rev. 2009, 2: CD007146-

    PubMed  Google Scholar 

  39. Cameron ID, Murray GR, Gillespie LD, Robertson MC, Hill KD, Cumming RG, Kerse N: Interventions for preventing falls in older people in nursing care facilities and hospitals. Cochrane Database Syst Rev. 2010, 1: CD005465-

    PubMed  Google Scholar 

  40. Meho LI: The rise and rise of citation analysis. Phys World. 2007, 20 (1): 32-36.

    Article  Google Scholar 

  41. Bunn F: The impact of systematic reviews on health care policy in England. 2010, PhD dissertation: University of Hertfordshire

    Google Scholar 

  42. The Cochrane Library 2010 Impact Factor. Cochrane database of systematic reviews (CDSR). Frequently asked questions. [http://www.thecochranelibrary.com/SpringboardWebApp/userfiles/ccoch/file/Impact%20Factor/FAQ%20for%202010%20IF.pdf],

  43. Falagas ME, Pitsouni EI, Malietzis GA, Pappas G: Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2008, 22 (2): 338-342.

    Article  CAS  PubMed  Google Scholar 

  44. Cozzens SE: The knowledge pool: measurement challenges in evaluating fundamental research programs. Eval Program Plann. 1997, 20 (1): 77-89. 10.1016/S0149-7189(96)00038-9.

    Article  Google Scholar 

  45. Comroe J, Dripps R: Scientific basis for the support of biomedical science. Biomedical Scientists and Public Policy. Edited by: Hugh Fudenberg H, Melnick VL. 1978, US: Springer, 15-33.

    Chapter  Google Scholar 

  46. Becker L, Oxman AD: Chapter 22: overview of reviews. Cochrane Handbook for Systematic Reviews of Interventions Version 5.10 [updated March 2011]. Edited by: Higgins J, Churchill R, Cumpston M, Chandler J. 2011, Oxford: The Cochrane Collaboration, [http://www.cochrane-handbook.org],

    Google Scholar 

  47. Peckham S, Willmott M, Allen P, Anderson S, Goodwin N: Assessing the impact of the NHS Service Delivery and Organisation Research and Development Programme. Evid Pol: J Res Debate Pract. 2008, 4: 313-330. 10.1332/174426408X338857.

    Article  Google Scholar 

  48. Bunn F, Sworn K: Strategies to promote the impact of systematic reviews on healthcare policy: a systematic review of the literature. Evid Pol: J Res Debate Pract. 2011, 7 (4): 403-428. 10.1332/174426411X603434.

    Article  Google Scholar 

  49. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J: A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014, 14: 2-10.1186/1472-6963-14-2.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Gillies D, Maxwell H, New K, Pennick V, van der Wouden H, Fedorowicz Z, Verbeek J, Ciapponi A: Collaboration-wide survey of Cochrane authors. 2009, [http://www.cochrane.org/collaboration-wide-survey-of-cochrane-authors],

    Google Scholar 

  51. Ismail S, Nason E, Marjanovic S, Grant J: Bibliometrics as a Tool for Supporting Prospective R&D Decision-Making in the Health Sciences: Strengths, Weaknesses and Options for Future Development. 2009, RAND: Santa Monica

    Google Scholar 

  52. van Leeuwen TN, Moed HF, Reedijk J: Critical comments on Institute for Scientific Information impact factors: a sample of inorganic molecular chemistry journals. J Inf Sci. 1999, 25 (6): 489-498. 10.1177/016555159902500605.

    Article  Google Scholar 

  53. Sabatier PA: An advocacy coalition framework of policy change and the role of policy-oriented learning therein. Pol Sci. 1988, 21 (2–4): 129-168.

    Article  Google Scholar 

  54. Sheldon TA, Cullum N, Dawson D, Lankshear A, Lowson K, Watt I, West P, Wright D, Wright J: What’s the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patients’ notes, and interviews. BMJ. 2004, 329 (7473): 999-10.1136/bmj.329.7473.999.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Ryan J, Piercy J, James P: Assessment of NICE guidance on two surgical procedures. Lancet. 2004, 363 (9420): 1525-1526. 10.1016/S0140-6736(04)16180-1.

    Article  PubMed  Google Scholar 

  56. Thomas L, Cullum N, McColl E, Rousseau N, Soutter J, Steen N: Guidelines in professions allied to medicine. Cochrane Database Syst Rev. 2000, 2: CD000349-

    PubMed  Google Scholar 

  57. Grimshaw JM, Russell IT: Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993, 342 (8883): 1317-1322. 10.1016/0140-6736(93)92244-N.

    Article  CAS  PubMed  Google Scholar 

  58. Grimshaw J, Freemantle N, Wallace S, Russell I, Hurwitz B, Watt I, Long A, Sheldon T: Developing and implementing clinical practice guidelines. Qual Health Care. 1995, 4 (1): 55-64. 10.1136/qshc.4.1.55.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  59. Martin HT, Mark JD, Jamie MF, Corey GR, Vivian HC, David JC, Peter BL: Impact of the NICE guideline recommending cessation of antibiotic prophylaxis for prevention of infective endocarditis: before and after study. BMJ. 2011, 342: d2392-10.1136/bmj.d2392.

    Article  Google Scholar 

  60. Grimshaw J, Thomas R, MacLennan G, Fraser C, Ramsay C, Vale L, Whitty P, Eccles M, Matowe L, Shirran L: Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004, 8 (6): 1-72.

    Article  Google Scholar 

  61. Gold M: Pathways to the use of health services research in policy. Health Serv Res. 2009, 44 (4): 1111-1136. 10.1111/j.1475-6773.2009.00958.x.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to thank Emma Pinkney who worked on the study; Managing Editors at CRG editorial bases who contacted authors on our behalf; Mrs Linda Bose and Dr Vina Mayor, members of the Public Involvement in Research Group at the University of Hertfordshire, for their input on the study; Anne Eisinga, UK Cochrane Centre, who provided information on the inclusion of Cochrane reviews in NICE and SIGN guidance; and Gavin Stewart, Wiley, who provided data on publication and usage of Cochrane reviews.

Funding

This article presents independent research commissioned by the UK National Institute for Health Research (NIHR) under the Systematic Reviews Programme (Grant Reference Number 12-203-03). The views expressed in this paper are those of the authors and not necessarily those of the NHS, the NIHR, nor the Department of Health. The sponsor of the study had no role in study design, data analysis, data interpretation or writing of the report.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Frances Bunn.

Additional information

Competing interests

FB, PA, DT and SI are all authors on Cochrane reviews. FB and DT are editors of the Cochrane Injuries Group. LH and AM declare that they have no competing interests.

Authors’ contributions

FB, DT, PA and SI wrote the protocol. FB, DT, LH and AM analysed the data. FB wrote the first draft of the paper. PA, SI and DT contributed to the writing of the paper. All authors critically reviewed the manuscript and agreed the final version.

Electronic supplementary material

13643_2014_339_MOESM1_ESM.doc

Additional file 1: CRG questionnaire. Questionnaire for Cochrane Review Groups funded by NIHR HTA Systematic Review Programme. (DOC 82 KB)

Additional file 2: Data extraction form 1. Questionnaire data extraction form. (DOCX 14 KB)

Additional file 3: Author questionnaire. Questionnaire for authors of Cochrane Systematic Reviews. (DOCX 29 KB)

13643_2014_339_MOESM4_ESM.docx

Additional file 4: Data extraction form 2. CRIS Data extraction form for documentary and bibliometric analysis. (DOCX 28 KB)

13643_2014_339_MOESM5_ESM.docx

Additional file 5: Reviews selected for further analysis. Details of 60 reviews selected for further analysis. (DOCX 19 KB)

Additional file 6: Review impact. Summary of main impacts of 60 reviews selected for further analysis. (DOCX 20 KB)

13643_2014_339_MOESM7_ESM.docx

Additional file 7: Qualitative interview data. This table shows the themes that arose from our qualitative analysis and the evidence to support them. (DOCX 25 KB)

Authors’ original submitted files for images

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bunn, F., Trivedi, D., Alderson, P. et al. The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research. Syst Rev 3, 125 (2014). https://doi.org/10.1186/2046-4053-3-125

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/2046-4053-3-125

Keywords