The impact of Cochrane Systematic Reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research

Background There has been a growing emphasis on evidence-informed decision-making in health care. Systematic reviews, such as those produced by the Cochrane Collaboration, have been a key component of this movement. The UK National Institute for Health Research (NIHR) Systematic Review Programme currently supports 20 Cochrane Review Groups (CRGs). The aim of this study was to identify the impacts of Cochrane reviews published by NIHR-funded CRGs during the years 2007–2011. Methods We sent questionnaires to CRGs and review authors, interviewed guideline developers and used bibliometrics and documentary review to get an overview of CRG impact and to evaluate the impact of a sample of 60 Cochrane reviews. We used a framework with four categories (knowledge production, research targeting, informing policy development and impact on practice/services). Results A total of 1,502 new and updated reviews were produced by the 20 NIHR-funded CRGs between 2007 and 2011. The clearest impacts were on policy with a total of 483 systematic reviews cited in 247 sets of guidance: 62 were international, 175 national (87 from the UK) and 10 local. Review authors and CRGs provided some examples of impact on practice or services, for example, safer use of medication, the identification of new effective drugs or treatments and potential economic benefits through the reduction in the use of unproven or unnecessary procedures. However, such impacts are difficult to objectively document, and the majority of reviewers were unsure if their review had produced specific impacts. Qualitative data suggested that Cochrane reviews often play an instrumental role in informing guidance, although a poor fit with guideline scope or methods, reviews being out of date and a lack of communication between CRGs and guideline developers were barriers to their use. Conclusions Health and economic impacts of research are generally difficult to measure. We found that to be the case with this evaluation. Impacts on knowledge production and clinical guidance were easier to identify and substantiate than those on clinical practice. Questions remain about how we define and measure impact, and more work is needed to develop suitable methods for impact analysis.


Background
In recent years, there has been a growing emphasis on the use of evidence to inform decision-making in health care [1][2][3], with the use of evidence seen as particularly relevant to commissioning because of the large financial commitments involved and because of the increasing complexity of health-care management decisions [4]. In addition, there has been growing interest in the way in which research is used, with researchers increasingly expected to consider the wider impacts of their work [5]. This may include the contributions research makes to health, society, culture, the economy, quality of life and public policy.
One of the key aspects of evidence-informed policy and practice has been the development of methods for the synthesis and integration of primary research, in the form of systematic reviews. Systematic reviews have been regarded as particularly important tools for decision-makers as it inherently makes sense for decisions to be based on the totality of evidence rather than a single study [6,7]. However, despite this, the extent to which policy makers and practitioners use systematic reviews as a source of evidence has been questioned [4,6]. Indeed, it has long been recognised that research may not always have the impact that researchers desire [8,9].
The Cochrane Collaboration is an independent, international organisation involved in preparing, maintaining and disseminating systematic reviews evaluating the effectiveness of health-care interventions. Cochrane systematic reviews should be uniquely placed to influence policy, practice and research as they provide a comprehensive, critical summary of what is known about effectiveness on a given topic; the rigour of their methods is widely acknowledged, and they are periodically updated in light of new evidence. However, whilst it is acknowledged that Cochrane groups produce high-quality systematic reviews [10][11][12], there is at present a lack of information about the impacts of Cochrane reviews.
The UK National Institute for Health Research (NIHR) systematic review programme currently supports 20 Cochrane Review Groups (CRGs) that have their editorial bases in academic or health institutions in the UK. These groups cover a broad range of health-care areas and produce almost half of all Cochrane reviews. This study was commissioned by the NIHR to identify the impacts and likely impacts of Cochrane reviews published by the 20 NIHR-funded CRGs between the years 2007 and 2011. The aims of this study were to identify impacts on clinical guidance and clinical practice and identify important gaps in knowledge and possible influence on the conduct of new primary research studies. In addition, we sought to identify barriers and facilitators to Cochrane reviews being used by guideline developers.

Methods
We used a mixed methods approach informed by theories about research impact [9,[13][14][15] and guided by a framework that draws on previous work around the evaluation of research impact [16][17][18][19]. This framework, including main and subcategories, can be seen in Table 1. A variety of methods exist for evaluating research impact including bibliometrics, documentary analysis, semi-structured interviews, case studies and surveys [20,21]. As there are advantages and disadvantages of each method, it is generally recommended that a variety of sources to identify research impact are used [22,23]. We used a mixture of bibliometrics, documentary analysis, questionnaire surveys and interviews. Obtaining the 'insider account' has been recommended when evaluating research impact [21], and it was envisaged that staff at editorial bases and review authors would be important sources of information about review impact.
There were two main components to the study. In the first (phase 1), we aimed to obtain a general overview of the impact of the outputs produced by NIHR-funded CRGs, and in the second (phase 2), we undertook a more detailed evaluation of a sample of Cochrane reviews. The study was conducted between April and September 2013 and only included reviews first published or updated during the years 2007-2011 (time period was stipulated by the funder). The evaluation covered the following CRGs: Airways; Bone, Joint and Muscle Trauma; Cystic Fibrosis and Genetic Disorders Group; Dementia and Cognitive Improvement; Depression, Anxiety and Neurosis; Ear, Nose and Throat; Epilepsy; Eyes and Vision; Gynaecological Cancer; Heart; Incontinence; Injuries; Neuromuscular; Oral Health; Pain, Palliative and Supportive Care; Pregnancy and Childbirth; Schizophrenia; Skin; Tobacco Addiction; and Wounds. For an overview of study methods, including how each part informs the findings, see Figure 1.
Ethical approval for the study was obtained from the University of Hertfordshire Health and Human Sciences Ethics Committee with delegated authority (ECDA), reference number HSK/SF/UH/00003. We sent a questionnaire survey (Additional file 1) to the editorial base (n = 20) of each NIHR-funded CRG. The questionnaires were sent via email with a personalised covering letter explaining the purpose of the study. The survey included questions about general impact (both actual and potential) of the CRG output between the years 2007 and 2011; the questions were informed by our evaluation framework. Respondents were asked, where possible, to provide supporting evidence of impact. This could include citation in clinical guidelines, impact on practice (for example, changes to clinicians behaviour, changes to service organisation and delivery), or influence on future primary research. CRGs were also asked to identify reviews published (or updated) in the last 5 years that they considered to have had the most impact on policy and practice.

Documentary review and analysis of existing sources
In addition to the questionnaires, we hand-searched the annual reports that CRGs had provided to the NIHR Health Technology Assessment (HTA) programme for the years 2007-2012, reviewed data on the use of Cochrane reviews in National Institute for Health and Care Excellence (NICE) and Scottish Intercollegiate Guidelines Network (SIGN) guidance compiled by the UK Cochrane Centre and searched the Cochrane Quality, Innovation, Productivity and Prevention (QIPP) topics database via NHS Evidence (https://www.evidence.nhs.uk/qipp). This documentary analysis was supplemented with keyword searches on a number of databases and internet sites including Google, NHS Evidence, and the World Health Organisation (WHO).

Data verification and analysis
Data from the questionnaires were entered on an Excel spreadsheet. One researcher extracted examples of actual or potential impact for each CRG and recorded this on a specially designed data extraction form based on the categories from our framework (Additional file 2). Additional data identified via documentary review was added to this form. Data relating to guidelines were collated in a separate Excel spreadsheet which was stratified by CRG. Data, and how to interpret it, was discussed with a second member of the team. Information was critically assessed and, where possible, evidence sought to verify it. For example, when a review was said to have led to follow-on research, we searched for the study protocol or final report in order to check that the review had been cited as justification for the research, and we obtained hard copies of guidelines and searched using the word 'Cochrane' to verify which

Qualitative interviews
We undertook telephone interviews with a convenience sample of guideline developers; this included those involved in developing and managing guidelines and systematic reviewers and technical advisors producing evidence reviews to underpin guidance. In the UK, participants were recruited from NICE (or one of their collaborating centres) or SIGN, and international participants were recruited via the WHO. Potential participants were identified by one of the authors with further snowballing as required. Our approach was iterative, and recruitment was stopped when we felt that we had reached data saturation. We used a semi-structured interview schedule which included questions about how systematic reviews were used in the development of guidance, experiences of using Cochrane reviews and ways in which use of Cochrane reviews might be facilitated. Interviews lasted between 20 and 40 min and were taped and transcribed. Informed consent was obtained from each participant.

Analysis
We drew on thematic content analysis to enable key features of guideline developers' experiences of using Cochrane reviews to be elicited from the data [24]. To guarantee a degree of inter-rater reliability and transparency, two researchers independently scrutinised each transcript and applied open codes to text. From this, a list of initial codes and themes were created. This was refined after further discussion with the wider project team, and supporting evidence in the form of quotes was documented.
Methods for phase 2: evaluation of impact of representative sample of Cochrane reviews (2007)(2008)(2009)(2010)(2011) In phase 2, we undertook further analysis on a purposive sample of 60 Cochrane reviews published or updated between 2007 and 2011. For each CRG, we chose one review randomly and two on the basis that they had potentially had an impact. Where possible, we used the information provided by CRGs in their questionnaires to inform the selection of reviews. However, in some cases, CRGs did not provide this information and we used other sources, such as citation counts or data from Wiley (publishers of the Cochrane Library), to guide our selection. As impact may take some time to build, we weighted our sample towards those reviews published between the years 2007 and 2010.

Questionnaire survey with systematic review authors
We sent a questionnaire survey (Additional file 3) to first authors of all 60 reviews. Questions were similar to those in the CRG questionnaire with authors asked to consider impact on knowledge production and possible impact of the review on health policy and practice. Authors were also asked if they thought their review identified important gaps in knowledge and/or if it had any influence on the conduct of new primary research. Methods for analysis and data verification were similar to those for the CRG questionnaire. Non-responders received a second mailing from the researchers and, in addition, 26 authors were followed up by the Managing Editors at CRG editorial bases.

Documentary and bibliometric analysis
We undertook citation analysis in Web of Science (WoS), Scopus and Google Scholar (searches conducted May/June 2013); undertook searches on Google, NHS Evidence and TRIP ( http://www.tripdatabase.com/) using review author and title keywords (between May and July 2013); and reviewed data from the publishers of The Cochrane Library (Wiley) on review downloads and media mentions. We also used an alternative metric measure (http://www.altmetric.com/). This provides article level metrics which give an indication of the impact of a publication by looking at activity surrounding the publication on social media sites and in policy documents and newspapers. Articles for which no mentions have been recorded score 0. Results from the documentary and bibliometric analyses were entered into a specially designed data extraction form in Access (see Additional file 4). Data on citation counts was entered by one researcher and checked by a second.

Results
A total of 3,187 new and updated reviews were published on the Cochrane Database of Systematic Reviews between 2007 and 2011, 1,502 (47%) of which were produced by the 20 CRGs funded by the NIHR (see Figure 2).
Seventeen of the 20 (85%) CRGs and 29 (48%) of review authors returned questionnaires. There was some variation in the number of author questionnaires returned for each CRG with all three questionnaires returned for some groups and none for others. Of the 60 reviews selected for analysis, 9 (15%) were updates and the rest were new reviews. Thirteen (22%) were published in 2007, 23 (38%) in 2008, 12 (20%) in 2009 and 6 (10%) in 2010 and 2011, respectively. Thirty-four (57%) of the first authors were based in the UK. Details of these reviews, including title, country of first author and whether they were a new review or an updated review, can be seen in Additional file 5.
Knowledge production, identifying gaps in the evidence and stimulating research CRG and author questionnaires provided 40 examples where they felt reviews had influenced primary research, and 13 (22%) of the sample of 60 reviews had been cited in a protocol or the background of a primary research study. In general, most of these examples related to work conducted by the Cochrane reviewers themselves; respondents to the questionnaires were less sure if their review had influenced the research of others. Most of the examples of follow-on research were RCTs. A summary by CRG can be seen in Table 2.
Although there was considerable variation between the reviews, the data do suggest that many of the 60 reviews have been of interest to decision makers. For example, 27 (45%) of the 60 reviews had had 100 or more citations in Google Scholar and 5 had received over 400 citations. Citation counts were higher in Google Scholar than in WoS or Scopus. The number of downloads from the Cochrane Library varied considerably between reviews. Of the sample of 60 reviews, the ten that were downloaded most frequently (full text and abstract) for the years 2009-2011 can be seen in Figure 3. This figure give an indication of the impact of reviews within the research and practice communities and show how downloads for reviews have increased over the 5-year period.
Thirty-six (60%) of the 60 reviews had an Altmetric score of 1 or more with 12 (20%) having a score over 10, and 4 (7%) scoring over 50. Currently, mid-tier publications might expect 30%-40% of papers to have a score of at least 1 [25]. Although these scores need to be interpreted cautiously, for example, they may not accurately reflect the interest in reviews published before 2011, they do provide some indication of the interest around a review and the way Cochrane reviews may have impacted on knowledge production by stimulating discussion and debate [26]. Citations counts and Altmetric scores for each review can be seen in Table 3 with a summary of citation analysis data in Table 4.

Informing policy development
Systematic reviews from all the 20 CRGs were cited in some form of clinical or practice guidance. Across the CRGs, there were 722 citations in 248 guidelines (or in the evidence reviews used to develop the guidance) with 481 systematic reviews being cited at least once. Of these, 62 were international guidance, 175 national guidance, and 10 local guidance (e.g. at Trust or hospital level). Of the national guidance, 87 were developed in the UK, with Cochrane reviews cited in 30 sets of NICE guidance and 23 sets of SIGN guidance. Information on inclusion in guidelines (by CRG) is summarised in Table 5.
We interviewed eight participants, four from NICE (or NICE collaborating centres), two from SIGN and two from the WHO. Five of the interviewees were female and three were male. Two participants were senior managers involved at a strategic level in co-ordinating guideline development, and six were involved in the production of individual guidelines either as managers, systematic reviewers or technical advisors. More details of the participants and their roles can be seen in Table 6. Analysis of the interview transcripts resulted in six overarching themes and a number of subthemes relating to the views and experiences of guideline developers and their use of Cochrane reviews. The overarching themes are the following: the process of using Cochrane reviews in the development of guidance, the quality of Cochrane Reviews, culture and approaches, up-to-date evidence, methodological issues and collaboration and communication. These themes and subthemes can be seen in Table 7. Whilst a number of the issues that arose might be applicable to systematic reviews in general, the focus of our questions and analysis was on the use of Cochrane reviews.
Results from the semi-structured interviews suggest that searching for relevant Cochrane reviews is part of the guideline development process and that Cochrane reviews often play an instrumental role in informing guidance. Cochrane reviews appeared to be used at a number of different stages of the guideline development process, for example, being used early in the process to scope review questions and assess the strength of the evidence and later in the process as part of the evidence review to develop the guidance. Even when the whole Cochrane review was not used, guideline developers often drew on component parts of the review such as search strategies, lists of included and excluded studies, quality assessment data and analyses. However, there were a number of barriers to the use of Cochrane reviews in guidance. Cochrane reviews might not be available, they might not fit with the guideline scope, they may be out of date, or the methods used may not fit with those required for the guideline (see Table 7). Evidence to support the thematic analysis can be seen in Additional file 6.

Impact on clinical practice and services
There was evidence to suggest that some Cochrane reviews may have contributed to a number of benefits to the health service including safer or more appropriate use of medication or other health technologies or the identification of new effective drugs or treatments (see the list below). Eight CRGs and 12 authors gave examples of impact on practice or services. However, it was difficult to verify some of these impacts and to judge whether these changes were directly attributable to the Cochrane review(s). Eighty-three percent of review authors who responded to the questionnaire were unsure if their work had changed the behaviour of practitioners, managers or members of the public, or if their work had helped reduce costs (69%), increase quality (76%), improve effectiveness (76%) or promote equity (79%).
A review on support surfaces for pressure ulcer prevention [27] was used to inform guidance on purchasing within the NHS [28]. Reviews on long-acting beta-antagonists (LABA) in asthma [29][30][31]    stop starch use within the NHS, a decision that has the potential to save both lives and money [34]. An updated review on antiviral treatment for Bell's palsy [35] may have contributed to changes in practice and a reduction of prescriptions of antiviral drugs for Bells palsy (http://cks.nice.org.uk/bells-palsy). A review on antifibrinolytic drugs for trauma patients [36,37] led to follow-on research which influenced the decision by the Medicines Innovation Scheme to fast track tranexamic acid for use in the NHS. Ambulance crews throughout the NHS now administer tranexamic acid to bleeding trauma patients (http://www.swast.nhs.uk/txa.htm).
Potential benefits of Cochrane reviews were highlighted in the NICE Cochrane Quality and productivity topics, 19 of which related to reviews produced by one of the 20 CRGs during the years 2007-2011. Potential benefits identified included economic benefits through budget savings or the release of funds, improvements in clinical quality, the reduction in the use of unproven or unnecessary procedures and improvements in patient and carer experiences.

Summary of findings
There was evidence that reviews from all the CRGs had had some impact. The clearest impacts were on healthcare policy, with less evidence of direct impact on research targeting, clinical practice and the organisation and delivery of NHS services. From our sample of 60 reviews, there was evidence to suggest that some reviews have had a significant impact on the research and practice communities, whilst others appear to have had little or no impact (a summary of the main impacts of the 60 reviews can be seen in Additional file 7). Some of the reviews associated with the clearest evidence of impact had been updated during 2007-2011 rather than first published during that time [33,38,39].
There was considerable variation in evidence of impact between CRGs and between reviews. Variation between CRGs might be accounted for by differences in the scope or speciality of the group, the type and number of outputs, or methods used for dissemination and knowledge transfer. However, it may also be a reflection of whether CRGs routinely collect data on impact. Variation amongst the sample of 60 reviews might be a reflection of the relevance of the review findings to decision makers, the date of  publication, the strength of the evidence or the strategies used to disseminate the findings.

Knowledge production and research targeting
We found evidence to suggest that some Cochrane reviews had played a role in identifying gaps in the evidence and stimulating new research. Across the CRGs, there were 40 examples of reviews influencing primary research. However, most of these examples of research impact related to work conducted by the Cochrane reviewers themselves; there was less evidence of a broad impact on stimulating new research. Based on the assumption that influential or important work will be cited more frequently than others [40] and that alternative metrics give an indication that papers have been read and discussed [26], the results of the bibliometric analysis suggest that a number of the reviews had had an impact on the creation of new knowledge and the stimulation of discussion and debate. Citation counts were much higher in Google Scholar than in WoS or Scopus. Google Scholar may be of particular importance to citation analyses for Cochrane reviews as previous work [41,42] suggests that citation counts for Cochrane reviews are artificially low in ISI databases and Scopus because citing authors have incorrectly referenced Cochrane reviews. However, as there are also some concerns about the accuracy of Google Scholar [43], evaluations of citation data for Cochrane reviews should include more than one database.

Informing policy development
Reviews from all the CRGs were cited in some form of clinical or practice guidance. Mostly, this was national and international guidance, with only ten examples of local guidance. This may be because local guidance is often not available outside of the organisations involved and so may be more difficult to find. CRGs and review authors gave us a number of anecdotal examples of reviews influencing local guidance (for example, at hospital or department levels), but most were excluded from   Table 7 Results of thematic analysis and barriers and facilitators to the use of Cochrane reviews in the development of guidance Themes and subthemes Barriers and facilitators Theme 1. The process of using Cochrane reviews (CRs) in the development of guidance • CRs used early in process/used in development phase Barriers • Systematic reviews top of evidence hierarchy/priority over other forms of evidence • CRs may not be available, may not fit with guideline scope • Guideline developers (GD) will use CR if available, but not always possible-CR may not be available/may not 'fit' • CR may be out of date • GD may use whole CR or parts of CR (e.g. using evidence tables)/ parts used vary

Facilitators
• CRs can save GD time (e.g. using existing searches/data) • Similar evidence hierarchy • GD may build on work of Cochrane reviewers/existing reviews • Cochrane processes for searching/ identifying studies seen as reliable and thorough • GD may redo the review (depending on resources) • Similar processes for critical appraisal • Lack of clarity on which follow-up data used from papers • Need for network meta-analysis and comparative analysis reviews our final numbers as we were unable to find a copy of the guidance in order to verify that it referenced a Cochrane review. It is possible, therefore, that the impact of Cochrane reviews on local guidance is underestimated in our evaluation. The fact that a review is cited in a guideline does not mean that it was instrumental in the development of the final guidance. Conversely, a Cochrane review might have been used to inform policy development but may not be referenced or cited. Interviews with guideline developers suggest that Cochrane reviews are routinely used to inform the guideline development process. However, reviews with a narrow focus or that are out of date are of less use to decision maker.

Impact on clinical practice and services
Review authors who responded to the questionnaire were generally unsure if their work had changed the behaviour of practitioners, managers or members of the public, or if their work had helped reduce costs, increase quality, improve effectiveness or promote equity. CRGs and authors did provide some examples to suggest that Cochrane reviews had contributed to a number of benefits to the health service including safer or more appropriate use of medication or other health technologies or the identification of new effective drugs or treatments, or that they had the potential to lead to cost-saving and health-service benefits. However, attributing particular behaviour changes, health benefits or cost saving to a particular systematic review (or reviews) is difficult. Generally, new research adds to an existing pool of knowledge [44] and many research projects may lie behind a specific advance in health care [45]. Many Cochrane reviews have a narrow focus, often deliberately. Whilst, this may ensure internal validity, it may reduce its impact in the policy and practice arenas, a factor recognised in the move by the Cochrane Collaboration to introduce overviews of reviews [46].

Strengths and limitations
Evaluating the impact of research is complicated. At present, there are no agreed instruments or methods for determining impact [47], and it is acknowledged that knowledge production is more easily discernible than impact on policy or health gain [21]. Moreover, although health benefits and broader economic benefits may be viewed as the real 'payback' from health research, these are hard to measure as it is difficult to attribute particular health gains to specific pieces of research [23]. Although we were able to make some inferences about health and economic benefits, these were largely beyond the remit of this study.
Our approach included the use of citation analysis and documentary review. Bibliometric techniques have been criticised for focusing on quantity rather than quality, and measuring the number of research outputs rather than research outcomes or impact [20]. Moreover, it may take several months or even years for a work to be • Formal links between CRG and guideline developers to promote use of CR • Financial support • GD experience problems communicating with CRGs • Issues of ownership/authorship-recognition and reward first cited. However, we combined traditional citation counts with the use of alternative metrics. This gave us another measure of the amount of attention a review had received including its impact in social media. In addition, our mixed methods approach, including documentary review, questionnaires with CRGs and review authors, and interviews with guideline developers, enabled us to get a more complete picture of impact. We interviewed only eight guideline developers, six of whom were based in the UK, and all of whom came from leading guideline development agencies. It is possible that their views and experiences may not be directly transferable to other organisations or to guideline developers in other countries. However, we found consistent and recurring themes in the interviews, and the views expressed are concordant with literature in this area [48,49].
We examined the impact of three reviews per CRG. Our sample is relatively small, and it is possible that the reviews we chose are not typical of the outputs of those CRGs, that there are better examples of impact and that the findings are not generalizable. Moreover, only 48% of the authors responded to the survey which raises further questions about the generalizability of the findings. However, this response rate compares favourably with previous surveys of Cochrane authors [50].
Although most of the data in the report are presented stratified by CRG, direct comparisons between reviews or between CRGs may not be appropriate. The diverse nature of the review topics and the differences in the number of questionnaires returned for each group and in length of time since publication mean that one should be cautious when comparing the impact of different reviews. Citation volume typically peaks in the third or fourth year postpublication, and therefore, a window of 5 years has been suggested as most appropriate for research assessment [51,52]. Reviews published more recently may not yet have had time to impact on policy or practice. Indeed, it has been suggested that it may take up to a decade for the full impact of research to be apparent [53].
There are several potential sources of bias in this evaluation. This was a retrospective analysis and as such may be at greater risk of bias than one where data is collected prospectively. In addition, much of our data was obtained from questionnaires to CRGs and review authors, and there is a risk of recall bias or that respondents might have inflated the impact of their work. In order to overcome this, the research team critically assessed all information provided and sought evidence to verify impacts and check whether it was related to an output produced in the time frame of interest.

Implications of the findings
Our study provides evidence that Cochrane reviews produced by NIHR-funded CRGs have an impact on clinical guidance development and may have influenced the conduct of primary research. We found limited evidence that Cochrane reviews had had a direct effect on clinical practice, but they may have an indirect impact on health care and patient outcomes through their role in informing further primary research and clinical guidance. Although the implementation of NICE guidance has been shown to be variable [54,55], there is evidence that clinical guidelines can be effective in changing the process and outcome of care [56][57][58][59][60].
There are significant difficulties associated with determining the impact of specific pieces of research [21]. The focus of this evaluation was on the outputs of 20 CRGs covering a broad range of health-care topics. It is possible that it would be easier to determine impact on clinical practice and the behaviour of health-care providers if such evaluations had a narrower focus. This might allow the use of more qualitative and quantitative methods targeted at specific groups of health-care providers.
Interviews with guideline developers suggest that Cochrane reviews are routinely used in the guideline development process. However, reviews with a narrow focus or that are out of date are of less use to decision makers. Use of Cochrane reviews in the development of guidelines might be facilitated by better collaboration and communication between CRGs and guideline developers. Formal as well as informal networks might be needed to facilitate the transfer of research knowledge to decision makers [61].