Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

Guidelines for randomized clinical trial protocol content: a systematic review

  • Jennifer M Tetzlaff1Email author,
  • An-Wen Chan2,
  • Jessica Kitchen2,
  • Margaret Sampson3,
  • Andrea C Tricco4 and
  • David Moher1
Systematic Reviews20121:43

DOI: 10.1186/2046-4053-1-43

Received: 26 March 2012

Accepted: 25 June 2012

Published: 24 September 2012

Abstract

Background

All randomized clinical trials (RCTs) require a protocol; however, numerous studies have highlighted protocol deficiencies. Reporting guidelines may improve the content of research reports and, if developed using robust methods, may increase the utility of reports to stakeholders. The objective of this study was to systematically identify and review RCT protocol guidelines, to assess their characteristics and methods of development, and to compare recommendations.

Methods

We conducted a systematic review of indexed literature (MEDLINE, EMBASE and the Cochrane Methodology Register from inception to September 2010; reference lists; related article features; forward citation searching) and a targeted search of supplementary sources, including a survey of major trial funding agencies in six countries. Records were eligible if they described a content guideline in English or French relevant to RCT protocols. Guidelines were excluded if they specified content for protocols for trials of specific procedures or conditions or were intended to assess trial quality. We extracted guideline characteristics and methods. Content was mapped for a subset of guidelines that described development methods or had institutional endorsement.

Results

Forty guidelines published in journals, books and institutional reports were included in the review; seven were specific to RCT protocols. Only eight (20%) described development methods which included informal consensus methods, pilot testing and formal validation; no guideline described all of these methods. No guideline described formal consensus methods or a systematic retrieval of empirical evidence to inform its development. The guidelines included a median of 23 concepts per guideline (interquartile range (IQR) = 14 to 34; range = 7 to 109). Among the subset of guidelines (n = 23) for which content was mapped, approximately 380 concepts were explicitly addressed (median concepts per guideline IQR = 31 (24,80); range = 16 to 150); most concepts were addressed in a minority of guidelines.

Conclusions

Existing guidelines for RCT protocol content varied substantially in their recommendations. Few reports described the methods of guideline development, limiting comparisons of guideline validity. Given the importance of protocols to diverse stakeholders, we believe a systematically developed, evidence-informed guideline for clinical trial protocols is needed.

Keywords

Randomized controlled trials Systematic review Protocols Clinical trials Reporting guideline SPIRIT initiative

Background

All randomized clinical trials (RCTs) require a protocol describing the rationale, methods, proposed analysis plan and organizational/administrative details from trial inception to reporting of results. Throughout a trial, diverse groups use the trial protocol, including investigators, participants and personnel, funding agencies, research ethics committees/institutional review boards (REC/IRB), journal editors and systematic reviewers. Transparent and clearly written protocols are important to guide trial conduct. They enable thorough assessment of adherence to scientific and ethical standards prior to trial inception [16] and monitoring of changes made throughout a trial that could bias interim or final trial results [5]. Some journals now require the submission of protocols with trial manuscripts, which are then included in the peer review process [710].

Unfortunately, a high proportion of trial protocols do not adequately describe important methodological details, decreasing their utility for trial implementation and critical appraisal of trials. For example, protocols often fail to designate primary outcomes [11] or detail allocation concealment [12], sample size calculations [13] and sponsor and investigator roles in trial conduct [14], all of which have been associated with biased trial results and conclusions. Additionally, comparisons of trial protocols with corresponding journal publications have consistently shown important, unacknowledged discrepancies, including discrepancies in primary outcomes [5] and statistical methods [13, 15]. With recent calls for greater access to trial protocols [16, 17] and trial registration [18], the content of trial protocols is receiving increased attention.

Reporting guidelines have been developed to improve the transparency of other research documents such as reports of research findings for journal publication [1926]. Indeed, the implementation and endorsement of some of these guidelines, including the CONSORT Statement (CON solidated S tandards O f R eporting T rials) [27] for reports of RCT findings, have been empirically shown to improve report quality [2830]. However, development methods of reporting guidelines vary, potentially impacting their utility to various stakeholders [20]. Some groups advocate that reporting guidelines should be developed using rigorous, systematic and transparent methodology and recommendations for reporting guideline development have recently been proposed [31].

Guidelines for clinical trial protocol content are available from varied sources, such as textbooks, funding applications and institutional guidelines. However, to our knowledge, their characteristics and methods of development have not been reviewed. In this paper we report a systematic review with the following objectives: 1) to identify reporting guidelines relevant to RCT protocols; 2) to examine their characteristics and development methods; and 3) to review their content.

Methods

The systematic review protocol was developed with input from trial and systematic review methodologists (See Appendix A in Additional file 1). This report describes the results of an updated review (original completed 2007; JT, AWC, ACT, DM; unpublished).

Eligibility criteria

Documents were eligible if they included a guideline that explicitly itemized content that should be included in protocols for human RCTs and that recommended methodological details beyond a generic heading for ‘Methods’; the term ‘proposal’ was added to the original eligibility criteria as it is sometimes used to refer to the protocol. Documents describing only common or typical protocol content (without recommending content) were excluded; when intent was unclear, the report was included. Tools were excluded if they recommended content specific to a narrow health care research area (for example, disease stage based on a specific classification system) as we intended to focus on guidelines that could be generalized to other research topics; to guide specific protocol aspects such as quality of life assessment; or to assess clinical trial quality. For practical reasons, guidelines were limited to those available in English or French. Both published and unpublished guidelines were eligible.

Information sources

Relevant guidelines were identified via two methods: 1) systematic review of indexed literature and 2) targeted search of major RCT funding agencies.

Systematic review of the literature

Searches were conducted in MEDLINE including in-process and other non-indexed citations (1948 to September Week 4 2010, Ovid interface); EMBASE including EMBASE Classic (1947 to 2010 Week 40, Ovid interface); and the Cochrane Methodology Register (The Cochrane Library 2010, Issue 4, Wiley interface). An information specialist (MS) developed the search strategies. The MEDLINE search, presented in Appendix B in Additional file 1, was amended for the other electronic databases (available upon request). Overlap in journal coverage between MEDLINE and EMBASE was removed using a Batch Citation Matcher technique [32], and remaining duplicates were removed in Reference Manager (Thomson Reuters Corporation, New York, USA). We also searched reference lists of included studies, the PubMed ‘related articles’ link, SCOPUS to identify reports citing the included studies, major clinical trials registries (clinicaltrials.gov, controlled-trials.com), and the EQUATOR Network (E nhancing the QUA lity and T ransparency O f health R esearch) database of reporting guidelines [19] for additional relevant guidelines. A sample of books was also reviewed and were identified based on book title through reference lists and via searches on Amazon.com [33], WorldCat.org [34], and local library portals using the search terms ‘protocols’ or ‘clinical trials’.

Targeted searching of RCT funding agencies

We reviewed a sample of guidelines from major clinical trial funding agencies as we expected some key guidelines would not be readily identifiable through electronic database searching. A short questionnaire was circulated by email to a convenience sample of six key informants who were nominated by the research team, each representing one of six countries previously identified as the top ‘health-related publication producers’ [35]: United States, United Kingdom, Japan, Germany, France and Canada. Informants provided up to two nominations from each of the following sectors within their country: major public (for example, governmental), non-governmental, non-profit (for example, charitable), and for-profit (for example, pharmaceutical industry) clinical trial funding organizations. We reviewed the funding agencies’ websites and contacted the agencies/companies by email to identify relevant guidelines.

Study selection and data extraction

Two reviewers independently screened titles and abstracts followed by potentially relevant full text articles using pre-defined eligibility criteria and pilot-tested forms. Disagreements were resolved by consensus or by the involvement of a third reviewer.

The following data were extracted from the included studies: report characteristics (publication year/version, source); guideline characteristics (format, intended scope, funding, and endorsement); contributors (number, country of residence and expertise of contributors); guideline development process (including use of consensus methods or evidence to inform content, pilot and validity testing, time-frame and dissemination methods); and guideline content (number and content of recommended items).

Two reviewers independently extracted data from a 25% random sample of studies using a standardized pilot-tested form (JT and JK). The remaining characteristics were then extracted by one reviewer, with key data (methods, search for evidence and number of guideline items) extracted in duplicate independently by a second reviewer. Where multiple reports of a guide or associated website were identified (that is, companion reports), all known associated sources were consulted and the most updated or complete version was treated as the primary source.

Guideline content

We itemized guideline content to compare recommendations across guidelines. This analysis was limited to evidence-informed guidelines, those with explicitly described methodology and those with either explicit or probable endorsement of the guideline by a recognized institution or organization. This subset was chosen to select guidelines that were potentially more rigorously developed (that is, those with methodology beyond the consensus of a few authors’ opinions) or more widely acknowledged. To aid in this comparison, we referred to the 2005 version (PR spreadsheet of elements V2.0; David Gemzik, personal communication) of Clinical Data Interchange Standards Consortium’s (CDISC) Protocol Representation Model [36], which aims to comprehensively list potential protocol concepts (to support the interchange of protocol information). Guideline content was mapped, where possible, to one of 264 concepts included in this model. Where no suitable concept existed or where the concept had a different level of granularity than the CDISC concepts, a new category was created. Content mapping was conducted by one reviewer (JT) and verified in full by a second reviewer (JK).

Synthesis of results

Data were summarized using descriptive measures. The following pre-specified sub-groups were compared descriptively: guidelines limited to RCTs versus those with a broader scope; guidelines with versus those without explicit development methods or cited evidence; and guidelines with versus those without described funding sources. A sensitivity analysis compared guidelines explicitly intended for ‘protocols’ versus those for ‘proposals’. Due to the nature of the review, no formal reporting bias assessments were conducted.

Results

Electronic searches yielded a total of 5,147 records (Figure 1), and 76 records were identified from other methods. Ten guidelines were identified from nominated funding agencies or their websites, eight of which were relevant; nine agencies confirmed no relevant guideline and no response was received (and no guideline located) for seven agencies (see Appendix C in Additional file 1 for list of agencies). After screening titles and abstracts, 384 full-text documents were reviewed and 46 were included in this review. Six guidelines were presented in two separate reports [3748], leaving 40 unique guidelines for data extraction [38, 40, 42, 44, 46, 4882] (Table 1).
https://static-content.springer.com/image/art%3A10.1186%2F2046-4053-1-43/MediaObjects/13643_2012_Article_44_Fig1_HTML.jpg
Figure 1

Flow of information through the systematic review.

Table 1

Guidelines included in this review (primary source)

Guideline

Reference

Institutional or named guidelines

 

A method for Rapid, Objective and Structured Evaluation (ROSE) of protocol of a randomised clinical trial (2005).

[56]

A Standard for the Scientific and Ethical Review of Trials (ASSERT) statement (2007).

[71]

Canadian Institutes of Health Research (2010 September) Funding opportunity details.

[75]

Centers for Disease Control and Prevention (2006) Developing a protocol: a guide for CDC investigators.

[69]

Chaput de Saintonge (1977) - produced by past and present members of Clinical Trials Unit, Department of Pharmacology and Therapeutics, London Hospital Medical College.

[55]

Clinical trial protocol templates at the National Institute of Allergy and Infectious Diseases (2009).

[59]

Code of Federal Regulations (2002): U.S. Food and Drugs. 21CFR312.23.

[82]

Code of practice for the clinical assessment of licensed medicinal products in general practice. (1983)

[63]

Guidelines for the preparation of E.R.T.C. cancer clinical trial protocols (1980).

[53]

International Conference on Harmonisation (1998). Harmonised Tripartite Guideline: Guideline for Good Clinical Practice E6 (ICH E6).

[70]

International Ethical Guidelines for Biomedical Research Involving Human Subjects (2002). Council for International Organizations of Medical Sciences (CIOMS) in collaboration with the World Health Organization (WHO)

[42]

Merck investigator studies program protocol template (2008). Requirements for submitting a full proposal.

[74]

National Health Service Department of Health [UK] (2011). Clinical trials toolkit - Based on ‘Detailed guidelines on the principles of good clinical practice in the conduct in the European Union of clinical trials on medicinal products for human use Ver 5.1’

[43, 44]

Pfizer Investigator-Initiated Research Program (2010). Investigator-initiated research & requests for pure substance (CTP).

[75]

Programm klinische studien leitfaden für die antragstellung (2010). Deutsche Forschungsgemeinschaft.

[81]

Schneidermann (1961) - Prepared for Cancer Chemotherapy National Service Center of the National Cancer Institute

[66]

The Trial Protocol Tool: The PRACTIHC software tool that supports the writing of protocols for pragmatic randomized controlled trials. (2006)

[46]

U.S. Department of Health and Human Services (2006). Public Health Service Grant Application (PHS 398).

[73]

Warren (1978) - produced by the South-east Thames Regional Health Authority's regional research committee (for which author was chairman)

[54]

Wellcome Trust (2011). Funding for clinical trials. Requirements for applicants.

[80]

Working group of the Commission on Dental Materials, Instruments, Equipment and Therapeutics (1977). Recommended outline for a research protocol.

[40]

Book chapters

[38, 48, 57, 58, 7780]

Other

[4952, 6062, 64, 65, 67, 68, 79]

Guideline characteristics and methods of development

We present the general characteristics of the guidelines in Table 2. The majority of guidelines were published as journal articles (n = 22; 55%); most were completed/published from 1991 to present (n = 24; 60%); and most were presented as checklists, tables or bullet lists (n = 27; 68%), some with additional explanatory text. Seven (18%) were specific to RCTs [46, 50, 51, 56, 65, 71, 75]. Fifteen guidelines were disseminated via websites (either specific to the tool or independent sites with links to view the guideline), and five reports described conference or lecture presentations [49, 51, 59, 62, 73]. Six (15%) guidelines reported a funding source, all of which were non-profit. Most guidelines (n = 25; 62%) had no clear institutional endorsement. Guidelines contained a median of 23 recommended items (IQR = 14, 34; range = 7 to 109).
Table 2

General characteristics of included guidelines (N = 40)

General characteristic

n (%)

Type of document

 

   Journal article

22 (55)

   Book

7 (16)

   Institutional or international agency guideline not presented in one of the above

10 (25)

   Other (1 website)

1 (3)

Date of publication/version

 

   Earlier than 1971

1 (3)

   1971-1980

5 (13)

   1981-1990

7 (18)

   1991-2000

6 (15)

   2001-2010

18 (45)

   Not stated/unclear

3 (8)

Format

 

   Text (for example, headings)

11 (28)

   Checklist/Table/Bullet list

16 (40)

   Checklist/Table/Bullet list and additional text

11 (28)

   Template

2 (5)

Scope

 

   RCT only

7 (18)

   More than RCTs

29 (73)

   Unclear/not stated

4 (10)

Guidance version

 

   New guidance

31 (78)

   Building on existing guidance

5 (13)

   Update of previous guidance

4 (10)

Funding

 

   Yes - non-profit

6 (15)

   Not reported/Unclear

33 (83)

   Reported no funding

0 (0)

Endorsement (explicit or probable)

15 (38)

Number of items - median (IQR); range

23 (14, 34); 7 to 109

Number of explicit concepts addressed* - median (IQR); range

31 (24, 80); 16 to 150

*from N = 23 guidelines with description of development methods, use of evidence to support recommendations, or institutional endorsement (explicit or probable).

In Table 3, we report aggregate results of guideline development methodology; details for each guideline are included in Appendix D in Additional file 1.
Table 3

Overview of guideline development methods (N = 40)

Characteristic

N (%)

Methods described (any)

 

   Yes

8 (20)

   None reported

19 (48)

   Unclear*

13 (33)

Consensus process

6 (15)

   Formal (for example, Delphi)

0 (0)

   Informal consensus and/or consensus meeting(s)

4 (10)

Pilot testing

2 (5)

Validation

 

   Formal validation

1 (3)

   Informal validation - shared with a broader circle of experts (for example, for face validity)

5 (13)

   Future plans to validate

1 (3)

Search for existing guidance/evidence to inform guideline recommendations

 

   Yes

2 (5)

     Search for previous guidelines

2 (5)

        Systematic

0 (0)

     Search for empirical evidence

1 (3)

        Systematic

0 (0)

   None reported

34 (85)

   At least one empirical study referenced to support at least one guideline item - methods of searching not reported

3 (8)

   Unclear

1 (3)

Number of authors/stated contributors - median (IQR); range

2 (1, 3); 1 to 88

   Not stated

11 (28)

Time frame for guideline development (years) [n = 4 guidelines] - median (IQR); range

4 (2, 4); 1 to 5

* If no methods were described, institutional guidelines were inferred to have some form of consensus and were labeled as unclear.

Eleven reports (28%) did not identify the contributors to their development. Of those that did, the majority (n = 24/29, 83%) listed at most 3 contributors (median (IQR) = 2 (1,3)) and most included contributors from a single country (n = 24/29; 83%); exceptions are noted in Appendix D in Additional file 1. Contributors’ areas of expertise were clearly reported for only 8 (20%) guidelines while other reports (for example, [43]) listed names and affiliations of all contributors without explicitly stating their areas of expertise. Contributors’ stated areas of expertise included clinical researchers/clinicians (five guidelines), methodologists/statisticians (two guidelines) and bioethicists, trial managers and information technology personnel (one guideline each).

Only eight reports (20%) described any development methods (Table 3 and Appendix D in Additional file 1): six journal articles [46, 5456, 59, 63] (two of which were specific to RCT protocols [46, 56]), and two international reports [42, 70]. Four of these guidelines describe more detailed and comprehensive methodology: the Council for International Organizations of Medical Sciences (CIOMS) International Ethical Guidelines for Biomedical Research Involving Human Subjects [42], the International Conference on Harmonization Tripartite Guideline for Good Clinical Practice E6 (ICH E6) [70], the PRACTIHC tool (Pragmatic Randomized Controlled Trials in HealthCare) [46] and templates developed for the United States National Institute of Allergy and Infectious Diseases [59] (See Appendix D in Additional file 1).

Stated development methods included informal consensus procedures (including consensus meetings) (n = 4; 10%) [42, 46, 59, 70]; pilot testing (n = 2; 5%) [46, 59]; soliciting input from a broader stakeholder group (for example, public/experts) prior to dissemination (n = 5; 13%) [42, 46, 54, 59, 70] and formal tool validation (n = 1; 3%) [56]. No report stated all of these methods. Two reports included a prospective request for public feedback after dissemination [55, 56] and one stated future plans for formal tool validation [59]. No report indicated the use of a formal consensus process (for example, Delphi consensus, Nominal Group Technique) for guideline development.

Additionally, no report described a systematic search for existing guidelines or empirical evidence to inform guideline content. One reported searching personal files [59] for previous guidelines and another reported a non-systematic search of the Internet and reference lists, and contacting experts to identify previous guidelines and evidence [46]. Three reports cited empirical evidence for some of the items [53, 58, 71] without describing methods for identifying this evidence.

Guideline content

We extracted content from a subset of 23 guidelines. The recommended content varied substantially between the guidelines (Table 4). Over 380 concepts were recommended (median (IQR) = 31 (24,80) concepts per guideline; range = 16 to 150), over half of which were each recommended in only one guideline (including both distinct concepts such as conflicts of interest and sub-concepts of existing headings such as rationale for choosing specific outcomes). We present the most commonly recommended concepts in Table 4.
Table 4

Common concepts requested in the 23 protocol guidelines with explicit methodology or institutional adoption (numbers in parentheses represent number of guidelines)*

Concepts in >75% guidelines

 

General

Statistical analysis

  Rationale/purpose (20)

  Statistical/analysis methods (general) (21)

  Objectives (general) (18)

 

Concepts in 51-75% of guidelines

 

General

Recruitment and eligibility (continued)

  Trial sites/institutions/ location (12)

  Sample size + sample size/power calculation (17)

  Background (general) (14)

Treatments and allocation

Design

  Treatment/interventions (general) (14)

  Study design type/description (17)

Assessments

Recruitment and eligibility

  Outcomes/ endpoints (list) (15)

  Eligibility criteria (16)

 

Concepts in 26-50% of guidelines

 

General

Statistical analysis

  Protocol/study title (9)

  Interim analysis description/general methods (7)

  Principal investigator - name and address (9)

  Trial termination criteria/stopping rules (6)

  Protocol summary (6)

Safety and monitoring

  Prior research (literature review) (10)

  Assessment of safety - general (7)

  Summary of known potential risks/benefits (6)

  Safety monitoring/procedures for unscheduled  events (9)

Design

  SAE reporting procedures (to sponsor and regulatory

  Statistical hypotheses (8)

  authorities) (8)

  Study schematic/flow-chart (for example, periods, duration) (6)

  Data monitoring committee (role, composition, independence) (7)

  Source population/recruitment source (7)

  Data management/record keeping (general) (9)

Recruitment and eligibility

  Methods to ensure data quality/integrity (for example, monitoring, validation) (7)

  Recruitment methods/subject selection (11)

Ethical considerations

  Information for patients and consent/methods (8)

  Ethics (general heading) (6)

  Eligible concomitant therapies (6)

  IRB review/approval/responsible IRBs (7)

  Subject drop-out criteria (7)

Dissemination

  Justification for sample size calculation (6)

  Publication policy (general) (6)

Treatments and allocation

  Disseminating results/publication plan (general) (7)

  Description of treatment (10) or comparators (8)

Other

  Treatment duration (8)

  Trial management (general - personnel and administration) (9)

  Allocation/randomization methods (general) (10)

  Budget (6)

  Degree of blind (for example, double, who)/blinding methods (8)

  References/cited literature (10)

Assessments

  Appendices (general) (6)

  General (variables and data collection) (10)

  Copy of all questionnaires and data forms (8)

  Primary endpoint(s)(7), Secondary endpoint(s) (6) (list)

 

  Methods of assessment/required tests (11)

 

  Timing of outcome assessment (7)

 

  Follow-up procedures (7)

 

Concepts in ≤ 25% of guidelines (n = 333; subset provided below)

 

General

Treatments and allocation (continued)

  Site investigators/collaborators - names and addresses (4)

  Reasons for degree of blinding (3)

  Sponsor - name and address (5)

Assessments

  Registration plans/registration number (4)

  Assessment of compliance (5)

  Protocol identifying number (5)

  Validity/reliability of collection/measurements (1)

  Funding source (3)

Statistical analysis

  Rationale with reference to a systematic review (3)

  Description of planned subgroup analyses (2)

Recruitment and eligibility

Other

  Target population (5)

  Participant security/confidentiality (5)

  Justification for special (for example, vulnerable) pop. (5)

  Study timetable (calendar (date) of events) (5)

  Specific eligibility criteria - for example, health status  (2), co-enrolment in trials (2)

  Target duration for trial as a whole (5)

Treatments and allocation

  Approximate time to complete enrolment (5)

  Treatment dosage (5), route of admin. (5), justification (5)

  Dummy tables (3)

  Form and location of treatment code (5)

  Curriculum vitae of investigators (5)

  Allocation concealment (3), Implementation of randomization (1)

  Incentive to investigators/staff (1), participants (2)

 

   Conflict of interest (1)

 

  Consumer involvement (who and roles) (1)

Subgroup comparisons

Few differences were noted between pre-specified subgroups by scope, development methods, and funding source. The number of guidelines in each subgroup was small, thus limiting the ability to make definitive conclusions. In Table 5 we present the most notable differences between the subgroups. No differences were found between guidelines intended for ‘protocols’ versus those for ‘proposals’.
Table 5

Major differences between pre-defined subgroups*

Methods

Recommendations

Guidelines specific to RCTs versus those with a broader scope (RCT + other study designs)

No differences

RCT only guidelines are more likely to recommend including:

 

· a systematic review as part of study background

 

· reason for degree of blinding

 

· methods to generate allocation sequence

 

· methods to implement randomization (who will generate sequence, who will enroll participants, who will allocate participants)

Guidelines with explicitly described methods of development versus those without

Those with explicit methods are more likely to:

Those with explicit methods are more likely to recommend including:

· describe multiple dissemination methods

· Procedures to break the blind

 

· Specific issues of consent with respect to vulnerable populations (for example, children, non-literate populations)

 

· Dosing frequency

 

· Duration of subject participation

Guidelines with explicit sources of funding versus those without

Funded guidelines are more likely to:

Guidelines with funding are more likely to recommend including:

· Have explicitly described methods (including informal consensus methods, searching for evidence, informal tool validation)

· Procedures to break the blind

 

· Specific issues of consent with respect to vulnerable populations (for example, children, non-literate populations)

· More likely to describe multiple dissemination methods

 
 

· Dosing frequency

· Have multiple contributors (Median [IQR] = 9 [3, 15]versus 1 [1, 3])

· Site representative/investigator

 

· Time and event schedules table

*Subgroups were compared descriptively; as sample sizes were small only major differences are noted. IQR: Inter-quartile range; RCT: randomized clinical trial.

Discussion

Our review identified numerous guidelines aiming to inform the content of clinical trial protocols. However, recommended concepts varied substantially across guidelines and the vast majority of guidelines did not describe their methods of development. When described, most included informal methods with limited stakeholder involvement and limited use of evidence to inform their recommendations. Similar findings have been reported elsewhere [20,][83].

Very few concepts were recommended consistently across guidelines, including several whose importance is supported by empirical evidence. For example, only half of the more recent guidelines [59, 70, 71, 74, 75, 80, 81] included an item recommending that primary outcomes be stated, despite preceding research showing biased modifications of primary outcomes throughout trials [3, 5, 11]. Similarly, only three [6, 66, 71] explicitly requested information regarding allocation concealment, the absence of which has been associated with inflated trial effect sizes [8486], although many requested general allocation methods. Conflicts of interest and roles of the sponsor in the trial were explicitly recommended in only one guideline [81], despite being required by the Declaration of Helsinki [87] and despite research showing that trials with financial competing interests report positive results more often than other trials [6, 88, 89]. Only three [71, 75, 81] guidelines explicitly recommended including or citing a systematic review as part of the trial rationale despite the problems associated with non-systematic literature searches [90, 91]. Finally, only 4 [46, 71, 72, 81] of 15 guidelines published after the introduction of trial registration requirements in 2005 [18] specifically requested registration information. No guideline recommended all of these important concepts.

The reasons for the variation and omissions are unclear. Few of the guideline reports in our sample described their development methods, preventing assessment of the validity of the recommendations. If not properly developed, guidelines could potentially ultimately be of limited use and may not improve the reporting of elements that are important to key users of protocols. Of the eight guidelines that did detail methodology, four seem relatively comprehensive [42, 46, 59, 70]. Although these four shared many common elements, considerable variation in recommended content was also present.

For a guideline to be widely acceptable, we believe it should be developed using robust methodology that engages key stakeholders during development and is guided by empirical evidence, where possible. In addition, the methodology should be clearly reported and accessible to enable understanding of the process and assessment of its validity. Recommendations for reporting guideline development have recently been proposed [31] and include a series of steps akin to those recommended for clinical practice guideline development [92]: involvement of multidisciplinary expert panel for a formal consensus process (for example, Delphi consensus) and consensus meeting(s), literature reviews to identify key evidence, pilot testing, active dissemination and impact evaluation. Recent research conducted by the EQUATOR group on the development of health care reporting guidelines [20, 83] suggests that such extensive methods are rarely employed. This is congruent with our current findings.

This review has some limitations. Although comprehensive in searching indexed periodicals, our review was not exhaustive in the search for institutional guidelines or books. However, our main findings would not likely substantively change with the inclusion of guidelines from these sources, as most guidelines available outside of journal articles did not describe development methods. Our results are also based on the methodology stated in included reports; we did not contact authors for additional information. Finally, the process of mapping and comparing concepts across guidelines was challenging due to the varied terminology used and the many sub-concepts of general headings that were recommended. To decrease bias we employed a systematic method and a second reviewer verified the process.

Our systematic review highlights some potential limitations of existing clinical trial protocol content guidelines. Given the evidence of protocol deficiencies [1114] and the importance of trial protocols to diverse stakeholders we believe there is a need for standard guidance that is developed using rigorous methods, broad consultation with key stakeholders and is based on empirical evidence, where possible. Development of reporting guidelines requires substantial resources and time [31], and the conduct of this review is an important inaugural step to justify undertaking such an initiative. Since the initial version of this review, an international collaboration known as the SPIRIT Initiative (S tandard P rotocol I tems: R ecommendations for I nterventional T rials) has convened to produce such guidance by systematically developing recommendations for minimum content of clinical trial protocols [93]. The primary aim of SPIRIT is to improve the content and utility of clinical trial protocols.

Conclusions

This review identified many guidelines for clinical trial protocols; the recommendations provided by these guidelines varied substantially and potentially important concepts were often not recommended. Most guidelines did not describe their methods of development and none of the reports described replicable methods of development including formal consensus of key stakeholders or a thorough search for relevant empirical evidence. Given the importance of trial protocols to diverse stakeholders and evidence of protocol deficiencies, we believe that development of a guideline meeting such standards is needed.

Financial disclosure

No direct funding was received for this study. Some authors were personally salaried by their institutions during the period of writing though no specific salary was set aside or given for the writing of this paper. Dr. Moher is supported, in part, by a University (of Ottawa) Research Chair. No funding bodies had any role in the study design, data collection, analysis, decision to publish or preparation of the manuscript.

Abbreviations

CDISC: 

Clinical Data Interchange Standards Consortium

CIOMS: 

Council for International Organizations of Medical Sciences

CONSORT: 

CONsolidated Standards Of Reporting Trials

EQUATOR: 

Enhancing the QUAlity and Tranparency Of Health Research

ICH E6: 

International Conference on Harmonization Tripartite Guideline for Good Clinical Practice E6

IQR: 

interquartile range

IRB: 

Institutional Review Board

PRACTIHC: 

Pragmatic Trials In Health Care

RCT: 

Randomized Clinical Trial

REC: 

Research Ethics Committee

SAE: 

Serious adverse event

SPIRIT: 

Standard Protocol Items, Recommendations for Interventional Trials.

Declarations

Acknowledgements

We extend our gratitude to Dr. Iveta Simera for sharing the initial report of a survey of guideline authors (now published [20]), influencing the development of the initial search strategy, screening criteria and data extraction forms for the current review. We thank Raymond Daniel for his time and effort towards running the searches, database management and article retrieval.

Authors’ Affiliations

(1)
Ottawa Methods Centre, Ottawa Hospital Research Institute
(2)
Department of Medicine, Women’s College Research Institute, University of Toronto
(3)
Library Services, Children's Hospital of Eastern Ontario
(4)
Li Ka Shing Knowledge Institute of St. Michael’s Hospital

References

  1. Gluud LL: Bias in clinical intervention research. Am J Epidemiol. 2006, 163: 493-501. 10.1093/aje/kwj069.View ArticlePubMed
  2. Bassler D, Ferreira GI, Briel M, Cook DJ, Devereaux PJ, Heels AD, Kirpalani H, Meade MO, Montori VM, Rozenberg A, Schunemann HJ, Guyatt GH: Systematic reviewers neglect bias that results from trials stopped early for benefit. J Clin Epidemiol. 2007, 60: 869-873. 10.1016/j.jclinepi.2006.12.006.View ArticlePubMed
  3. Chan A-W, Krleža-Jeric K, Schmid I, Altman DG: Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. Can Med Assoc J. 2004, 171: 735-740. 10.1503/cmaj.1041086.View Article
  4. Chan AW, Altman DG: Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005, 330: 753-10.1136/bmj.38356.424606.8F.PubMed CentralView ArticlePubMed
  5. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008, 3: 3081-10.1371/journal.pone.0003081.View Article
  6. Lexchin J, Bero LA, Djulbegovic B, Clark O: Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003, 326: 1167-1170. 10.1136/bmj.326.7400.1167.PubMed CentralView ArticlePubMed
  7. Lancet: Information for authors.http://download.thelancet.com/flatcontentassets/authors/lancet-information-for-authors.pdf,
  8. British Medical Journal: Resources for authors.http://resources.bmj.com/bmj/authors/article-submission/article-requirements,
  9. PLoS Medicine: Guidelines for Authors.http://journals.plos.org/plosmedicine/guidelines.php,
  10. Open Medicine: Author Guidelines.http://www.openmedicine.ca/about/submissions#authorGuidelines,
  11. Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004, 291: 2457-2465. 10.1001/jama.291.20.2457.View ArticlePubMed
  12. Pildal J, Chan AW, Hróbjartsson A, Forfang E, Altman DG, Gøtzsche PC: Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005, 330: 1049-10.1136/bmj.38414.422650.8F.PubMed CentralView ArticlePubMed
  13. Chan AW, Hróbjartsson A, Jørgensen KJ, Gøtzsche PC, Altman DG: Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols. BMJ. 2008, 337: a2299-10.1136/bmj.a2299.PubMed CentralView ArticlePubMed
  14. Gøtzsche P, Hróbjartsson A, Johansen H, Haahr M, Altman D, Chan A: Ghost authorship in industry-initiated randomised trials. PLoS Med. 2007, 4: e19-10.1371/journal.pmed.0040019.PubMed CentralView ArticlePubMed
  15. Hernández AV, Steyerberg EW, Taylor GS, Marmarou A, Habbema JD, Maas AI: Subgroup analysis and covariate adjustment in randomized clinical trials of traumatic brain injury: a systematic review. Neurosurgery. 2005, 57: 1244-1253.View ArticlePubMed
  16. Chan A-W: Bias, spin, and misreporting: Time for full access to trial protocols and results. PLoS Med. 2008, 5: e230-10.1371/journal.pmed.0050230.PubMed CentralView ArticlePubMed
  17. Chan A-W: Access to clinical trial data. BMJ. 2011, 342: d80-10.1136/bmj.d80.View ArticlePubMed
  18. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marušiæ A, Overbeke AJ, Schroeder TV, Sox HC, Van Der Weyden MB, International Committee of Medical Journal Editors: Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Croat Med J. 2004, 45: 531-532.PubMed
  19. Equator Network: Enhancing the QUAlity and Transparency Of health Research.http://www.equator-network.org,
  20. Simera I, Altman DG, Moher D, Schulz KF, Hoey J: Guidelines for reporting health research: the EQUATOR network's survey of guideline authors. PLoS Med. 2008, 5: e139-10.1371/journal.pmed.0050139.PubMed CentralView ArticlePubMed
  21. Campbell MK, Elbourne DR, Altman DG, CONSORT group: CONSORT statement: extension to cluster randomised trials. BMJ. 2004, 328: 702-708. 10.1136/bmj.328.7441.702.PubMed CentralView ArticlePubMed
  22. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, Oxman AD, Moher D: Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008, 337: 1223-1226.View Article
  23. Boutron I, Moher D, Altman DG, Schulz K, Ravaud P, for the CONSORT group: Methods and processes of the CONSORT Group: example of an extension for trials assessing nonpharmacologic treatments. Ann Intern Med. 2008, 148: W60-W66.PubMed
  24. Gagnier JJ, Boon H, Rochon P, Moher D, Barnes J, Bombardier C: Reporting randomized, controlled trials of herbal interventions: an elaborated CONSORT statement. Ann Intern Med. 2006, 144: 364-367.View ArticlePubMed
  25. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group: Preferred Reporting Items for Systematic reviews and Meta-Analyses: the PRISMA statement. PLoS Med. 2009, 6: e1000097-10.1371/journal.pmed.1000097.PubMed CentralView ArticlePubMed
  26. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP: STROBE Initiative: The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007, 370: 1453-1457. 10.1016/S0140-6736(07)61602-X.View ArticlePubMed
  27. Schulz KF, Altman DG, Moher D, CONSORT Group: CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. PLoS Med. 2010, 7: e1000251-10.1371/journal.pmed.1000251.PubMed CentralView ArticlePubMed
  28. Turner L, Moher D, Shamseer L, Weeks L, Peters J, Plint A, Altman DG, Schulz KF: The influence of CONSORT on the quality of reporting of randomised controlled trials: an updated review. Trials. 2011, 12 (Suppl 1): A47-10.1186/1745-6215-12-S1-A47.PubMed CentralView Article
  29. Smidt N, Rutjes AW, van der Windt DA, Ostelo RW, Bossuyt PM, Reitsma JB, Bouter LM, de Vet HC: The quality of diagnostic accuracy studies since the STARD statement: has it improved?. Neurology. 2006, 67: 792-797. 10.1212/01.wnl.0000238386.41398.30.View ArticlePubMed
  30. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006, 185: 263-267.PubMed
  31. Moher D, Schulz KF, Simera I, Altman DG: Guidance for developers of health research reporting guidelines. PLoS Med. 2009, 7: e1000217-View Article
  32. Sampson M, McGowan J, Cogo E, Horsley T: Managing database overlap in systematic reviews using Batch Citation Matcher: case studies using Scopus. J Med Libr Assoc. 2006, 94: 461463-
  33. Amazon.http://www.Amazon.com,
  34. WorldCat.http://www.WorldCat.org,
  35. Paraje G, Sadana R, Karam G: Public health. Increasing international gaps in health-related publications. Science. 2005, 308: 959-960. 10.1126/science.1108705.View ArticlePubMed
  36. Clinical Data Interchange Standards Consortium (CDISC): Protocol Representation Model.http://www.cdisc.org/protocol,
  37. Spilker B: Guide to clinical studies and developing protocols. 1984, New York: Raven
  38. Spilker B: Guide to clinical trials. 1991, New York: Raven
  39. Working group of the Commission on Dental Materials IEaT: Recommendation plan for research protocol. FDI Commission on Dental Products, Instruments, Equipment and Therapeutic Agents (COMIET), G.T. Clinical trials. The French version: INCOGUDET]. Int Dent J. 1977, 27: 299-300.
  40. Working group of the Commission on Dental Materials IEaT: Recommended outline for a research protocol. Prepared by a working group of the Commission on Dental Materials, Instruments, Equipment and Therapeutics. Int Dent J. 1977, 27: 44-45.
  41. Council for International Organizations of Medical Sciences: International ethical guidelines for biomedical research involving human subjects. Bull Med Ethics. 2002, 182: 17-23.
  42. Council for International Organizations of Medical Sciences (CIOMS) in collaboration with the World Health Organization (WHO): International Ethical Guidelines for Biomedical Research Involving Human Subjects. 2002, Geneva: Council for International Organizations of Medical Sciences (CIOMS)
  43. Detailed guidelines on the principles of good clinical practice in the conduct in the European Union of clinical trials on medicinal products for human use Ver 5.1.http://ec.europa.eu/health/files/pharmacos/docs/doc2002/july/gcp_51_july_en.pdf,
  44. NHS Department of Health [UK]: Clinical trials toolkit.http://www.ct-toolkit.ac.uk/_db/_documents/Protocol.pdf,
  45. Akl EA, Treweek S, Foy R, Francis J, Oxman AD: NorthStar, a support tool for the design and evaluation of quality improvement interventions in healthcare. Implement Sci. 2007, 2: 19-10.1186/1748-5908-2-19.PubMed CentralView ArticlePubMed
  46. Treweek S, McCormack K, Abalos E, Campbell M, Ramsay C, Zwarenstein M, PRACTIHC C: The Trial Protocol Tool: The PRACTIHC software tool that supported the writing of protocols for pragmatic randomized controlled trials. J Clin Epidemiol. 2006, 59: 1127-1133. 10.1016/j.jclinepi.2005.12.019.View ArticlePubMed
  47. Betkerur J: Guidelines for writing a research project synopsis or protocol. Indian J Dermatol Venereol Leprol. 2008, 74: 687-690. 10.4103/0378-6323.45136.View ArticlePubMed
  48. World Health Organization: Health research methodology: A guide for training in research methods. 2001, Manila: World Health Organization, Regional Office for the Western Pacific
  49. Indrayan A: Elements of medical research. Indian J Med Res. 2004, 119: 93-100.PubMed
  50. Girling DJ: Important issues in planning and conducting multi-centre randomised trials in cancer and publishing their results. Crit Rev Oncol Hematol. 2000, 36: 13-25. 10.1016/S1040-8428(00)00095-0.View ArticlePubMed
  51. Ivanov A, Van GM: Design, management and handling of a randomized trial. [French]. Bull Cancer. 1995, Suppl 5: 553s-557s.
  52. Moorhead JE, Rao PV, Anusavice KJ: Guidelines for experimental studies. Dent Mater. 1994, 10: 45-51. 10.1016/0109-5641(94)90021-3.View ArticlePubMed
  53. Staquet M, Sylvester R, Jasmin C: Guidelines for the preparation of E.R.T.C. cancer clinical trial protocols. Eur J of Cancer (Oxford). 1980, 16: 871-875.View Article
  54. Warren MD: Aide-memoire for preparing a protocol. Br Med J. 1978, 1: 1195-1196. 10.1136/bmj.1.6121.1195.PubMed CentralView ArticlePubMed
  55. de Saintonge CD: Aide-memoire for preparing clinical trial protocols. Br Med J. 1977, 1: 1323-1324.
  56. Malhotra S, Shafiq N, Pandhi P: A method for rapid, objective and structured evaluation (ROSE) of protocol of a randomised clinical trial. Bull, Postgrad Inst Med Ed Res, Chandigarh. 2005, 39: 22-26.
  57. Piantadosi S: Clinical trials: a methodologic perspective. 2005, Hoboken, N.J: Wiley-InterscienceView Article
  58. Pocock SJ: Clinical trials: a practical approach. 1983, New York: John Wiley and Sons Ltd, Chichester
  59. Bridge H, Smolskis M, Bianchine P, Dixon DO, Kelly G, Herpin B, Tavel J, Protocol Template Working Group of the NIAID Clinical Research Subcommittee: Development and implementation of clinical trial protocol templates at the National Institute of Allergy and Infectious Diseases. Clin. 2009, 6: 386-391.
  60. Laurencot CM, Ruppel S: Regulatory aspects for translating gene therapy research into the clinic. Methods Mol Biol. 2009, 542: 397-421.PubMed
  61. O'Dowd T, Jones S: Protocols: getting them right. Practitioner. 1988, 232: 713-714.PubMed
  62. Blandford DH, Warren GB, Campbell EM: Uses and content of a research protocol. J Dent Educ. 1984, 48: 298-301.PubMed
  63. British Medical Association, Royal College of General Practitioners and Association of the British Pharmaceutical Industry: Code of practice for the clinical assessment of licensed medicinal products in general practice. Br Med J (Clin Res Ed). 1983, 286: 1295-1297.View Article
  64. Souche A, Zekri J-R, Thermoz P: Therapeutic tests of the medicins a clinical protocol established on priority why?. Psychologie Medicale. 1990, 22: 1483-1484.
  65. Pignon JP, Doyon F: Randomized clinical trials. Part I. Principles, protocol and organization. Reanim Soins Intensifs Med D'Urgence. 1989, 5: 239-243.
  66. Schneidermann M: Controlled clinical trials: Monday's count-down for Tuesday's launching. J New Drugs. 1961, 1: 250-255.
  67. Bordage G, Dawson B: Experimental study design and grant writing in eight steps and 28 questions. Med Educ. 2003, 37: 376-385. 10.1046/j.1365-2923.2003.01468.x.View ArticlePubMed
  68. Williams AM: Neuroscience nursing: designing the research proposal. J Neurosci Nurs. 1989, 21: 389-391. 10.1097/01376517-198912000-00015.View ArticlePubMed
  69. Centers for Disease Control and Prevention: Developing a protocol: a guide for CDC investigators.http://www.umdnj.edu/hsweb/Forms/Forms_Developing_a_Protocol.pdf,
  70. ICH Harmonised Tripartite Guideline: Guideline for Good Clinical Practice E6.http://www.ich.org/cache/compo/276-254-1.html,
  71. A Standard for the Scientific and Ethical Review of Trials (ASSERT) Statement.http://www.assert-statement.org/explandoc.pdf,
  72. U.S. Department of Health and Human Services: Public Health Service Grant Application (PHS 398) Part I and Part II. OMB No. 0925,http://grants.nih.gov/grants/funding/phs398/phs398.html, -0001,
  73. Merck investigator studies program (MISP): Protocol template. Requirements for submitting a full proposal.http://merckresearch.net/doc/MISP%20Protocol%20Template.doc,
  74. Investigator-initiated research (IIR) & requests for pure substance (CTP).http://media.pfizer.com/files/research/iir_information.pdf,
  75. Funding opportunity details.http://www.cihr-irsc.gc.ca/e/39187.html,
  76. Schwartz D, Flamant R, Lellouch J: Clinical trials. 1980, New York - London: Academic
  77. Eschwège E, Bouvenot G, Doyon F, Lacroux A: Essais thérapeutiques: mode d'emploi. 1990, Paris, France: Inserm
  78. Bowling A: Research methods in health: investigating health and health services. 2002, Buckingham - Philadelphia: Open University Press
  79. Ahmed AM: A guide for writing a protocol for a clinical trial. Sudan J Public Health. 2010, 5: 23-28.
  80. Wellcome Trust: Funding for clinical trials. Requirements for applicants.http://www.wellcome.ac.uk/Funding/Biomedical-science/Application-information/WTX022708.htm,
  81. Programm klinische studien leitfaden für die antragstellung.http://www.gesundheitsforschung-bmbf.de/_media/Leitfaden_Klinische_Studien_100818.pdf,
  82. Food and Drug Administration: Code of Federal Regulations: Food and Drugs. 2002, 21CFR312.23
  83. Moher D, Weeks L, Ocampo M, Seely D, Sampson M, Altman DG, Schulz KF, Miller DL, Simera I, Grimshaw J, Hoey J: Describing reporting guidelines for health research: a systematic review. J Clin Epidemiol. 2011, 64: 718-742. 10.1016/j.jclinepi.2010.09.013.View ArticlePubMed
  84. Wood L, Egger M, Gluud LL, Schulz KF, Jüni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336: 601-605. 10.1136/bmj.39465.451748.AD.PubMed CentralView ArticlePubMed
  85. Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA, Schünemann H, Briel M, Nordmann AJ, Pregno S, Oxman AD: Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev. 2011, MR000012-
  86. Pildal J, Hróbjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gøtzsche PC: Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007, 36: 847-857. 10.1093/ije/dym087.View ArticlePubMed
  87. WMA Declaration of Helsinki - Ethical Principles for Medical Research Involving Human Subjects.http://www.wma.net/en/30publications/10policies/b3/index.html,
  88. Bekelman JE, Li Y, Gross CP: Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA. 2003, 289: 454-465. 10.1001/jama.289.4.454.View ArticlePubMed
  89. Kjaergard LL, Is-Nielsen B: Association between competing interests and authors' conclusions: epidemiological study of randomised clinical trials published in the BMJ. BMJ. 2002, 325: 249-10.1136/bmj.325.7358.249.PubMed CentralView ArticlePubMed
  90. Fergusson D, Glass KC, Hutton B, Shapiro S: Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding?. Clin. 2005, 2: 218-229.
  91. Puhan MA, Vollenweider D, Steurer J, Bossuyt PM, ter Riet G: Where is the supporting evidence for treating mild to moderate chronic obstructive pulmonary disease exacerbations with antibiotics? A systematic review. BMC Med. 2008, 6: 28-10.1186/1741-7015-6-28.PubMed CentralView ArticlePubMed
  92. Oxman AD, Fretheim A, Schunemann HJ, Haines A, Langer A, Pang T, Panisset U, Whitworth J: Improving the use of research evidence in guideline development: Introduction. Health Res Pol Syst. 2006, 2006: 4-
  93. Editorial: Strengthening the credibility of clinical research. Lancet. 2011, 375: 1226-

Copyright

© Tetzlaff et al.; licensee BioMed Central Ltd. 2012

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.