- Open Access
- Open Peer Review
The use of clinical study reports to enhance the quality of systematic reviews: a survey of systematic review authors
© The Author(s). 2018
- Received: 8 January 2018
- Accepted: 27 June 2018
- Published: 8 August 2018
Clinical study reports (CSRs) are produced for marketing authorisation applications. They often contain considerably more information about, and data from, clinical trials than corresponding journal publications. Use of data from CSRs might help circumvent reporting bias, but many researchers appear to be unaware of their existence or potential value. Our survey aimed to gain insight into the level of familiarity, understanding and use of CSRs, and to raise awareness of their potential within the systematic review community. We also aimed to explore the potential barriers faced when obtaining and using CSRs in systematic reviews.
Online survey of systematic reviewers who (i) had requested or used CSRs, (ii) had considered but not used CSRs and (iii) had not considered using CSRs was conducted. Cochrane reviewers were contacted twice via the Cochrane monthly digest. Non-Cochrane reviewers were reached via journal and other website postings.
One hundred sixty respondents answered an open invitation and completed the questionnaire; 20/160 (13%) had previously requested or used CSRs and other regulatory documents, 7/160 (4%) had considered but not used CSRs and 133/160 (83%) had never considered this data source. Survey respondents mainly sought data from the European Medicines Agency (EMA) and/or the Food and Drug Administration (FDA). Motivation for using CSRs stemmed mainly from concerns about reporting bias 11/20 (55%), specifically outcome reporting bias 11/20 (55%) and publication bias 5/20 (25%). The barriers to using CSRs noted by all types of respondents included current limited access to these documents (43 respondents), the time and resources needed to obtain and include these data in evidence syntheses (n = 25) and lack of guidance about how to use these sources in systematic reviews (n = 26).
Most respondents (irrespective of whether they had previously used them) agreed that access to CSRs is important, and suggest that further guidance on how to use and include these data would help to promote their use in future systematic reviews. Most respondents who received CSRs considered them to be valuable in their systematic review and/or meta-analysis.
The findings of clinical trials as reported in journal articles can sometimes be incomplete and misleading. There is evidence that analyses and outcomes, including both efficacy and harms, may be reported selectively such that the true effects of treatments remain hidden [1–5]. Consequently, those performing systematic reviews and meta-analyses may need to take additional steps to locate, appraise and synthesise missing or inadequately reported data in order to minimise the impact of such reporting bias.
Clinical study reports (CSRs) are produced by pharmaceutical companies during a marketing authorisation application for investigational medicinal products in the EU, Japan and the USA. They are usually written in accordance with the ‘international conference on harmonisation of technical requirements for registration of pharmaceuticals for human use’ (ICH) guideline on the structure and content of clinical study reports (ICH E3) . The purpose of the ICH guidance is to assist sponsors in developing a comprehensive trial report that is complete, well-structured and easy for regulators to review when making licencing decisions .
CSRs often include greater detail about trial design, conduct and analysis; more complete results; and a more reliable picture of strengths and weaknesses than journal articles. Extracting data from CSRs and using these in systematic reviews and meta-analyses may therefore provide more complete information and generate more reliable effect estimates [5, 8] than using data presented in journal articles and help circumvent reporting bias [9, 10], particularly in relation to adverse events [11–13].
CSRs are becoming increasingly available and accessible following liberalisation of the European Medicines Agency (EMA) policy related to public access to documents it holds relating to market access applications . Requests to the EMA for access to CSRs [15, 16] have increased from 20 requests per month during the first 2 years since 2010, to nearly 40 requests per month post the initial 2-year period. The shift towards improved transparency continued with the implementation of EMA’s Policy 0070 for publication of clinical data of medicinal products for human use , and other significant efforts to provide broader access to clinical trial data. These include the Yale University Open Data Access (YODA) Project , ClinicalStudyDataRequest.com (CSDR) , the Duke Clinical Research Institute (DCRI) , the AllTrials campaign  and OpenTrials.net . Some of the world’s largest pharmaceutical companies including Bristol-Myers Squibb, GlaxoSmithKline, Johnson & Johnson, Lilly and Roche have committed to data sharing . CSRs are now being promoted as a valuable resource in systematic reviews [5, 8, 9, 12], but because data sharing directly via companies and other various platforms is still relatively new, their existence may still be unknown to many systematic review authors, and their utility is still largely unexplored .
The US Agency for Healthcare Research and Quality (AHRQ) has, amongst others, noted the value of searching for ‘regulatory documents’ as a means of addressing reporting bias . The term ‘regulatory documents’ can be used to describe a number of sources of information other than CSRs, including Food and Drug Administration (FDA) approval documents on the Drugs@FDA website (e.g. medical and statistical reviewer reports), European Public Assessment Reports and any document produced by, or held by, a regulatory agency.
The current version of the Cochrane Handbook, last updated in 2011, encourages authors to search for unpublished data from pharmaceutical companies, trial registers and trial result registries . It does not, however, discuss searching for or considering the use of CSRs and other regulatory documents as sources of data, which could partly explain why there have so far been few Cochrane Reviews that have sought data from these sources [11, 27]. This will be addressed in the major revision currently in preparation—and is due for publication in late 2018.
Recognising the need to consider the potential value of CSRs and other regulatory documents as a potential source of data for Cochrane Reviews of pharmacological interventions, Cochrane funded a project (of which this survey was part) to explore the rationale for such use, and the readiness of Cochrane reviewers to engage with regulatory material. To assess readiness and raise awareness, we carried out a preliminary survey to gain insight to the level of understanding, familiarity with and views on the importance of CSRs and other regulatory documents. We also explored what has previously motivated authors to seek data from these sources, and barriers to using them in Cochrane and other systematic reviews. We then carried out a follow-up survey of respondents who had considered or used regulatory data in their systematic review, to explore under what circumstances they thought it most important to seek CSRs as a data source.
We conducted two online surveys involving authors of Cochrane and other systematic reviews using the data capture tool Qualtrics . The initial survey (Additional file 1) was open between 10 June and 19 September 2016. This was split into two releases, one intended for Cochrane authors and the other for authors of systematic reviews conducted outside of Cochrane. A second (follow-up) survey (Additional file 2) was open between 21 April and 31 May 2017.
The survey design included closed and open-ended questions. Response options were discussed, drafted and refined by the team. Question types included multiple choice (check one/check all), free-text and comments. ‘Other (specify)’ responses were offered to capture more detailed information that could not be collected using structured multiple choice questions. Pilot testing of the survey logic and flow was performed by AH and checked by members of the research team and four independent researchers. Revisions were made where necessary.
Ethics approval for the survey was granted by the University of York Health Sciences Research Ethics and Integrity Governance Committee.
The release of the initial survey intended for Cochrane review authors was first announced in the Cochrane Digest, which was emailed to all Cochrane authors on 10 June 2016. It was then mentioned again in the Cochrane Digest 2 weeks later. The release intended for authors of non-Cochrane reviews was first advertised on the University of York Centre for Reviews and Dissemination web site on the 25 June 2016 and then on the Systematic Reviews journal web site. Links to this were also shared via social media. The follow-up survey was sent only to those Cochrane respondents who had previously considered or used regulatory data in their systematic review and who had agreed during the first survey to participate in the follow-up. Although several authors of this manuscript have previously used data from regulatory documents, purposely none participated in either survey.
Domains of interest
‘Requested’ (i.e. had used data from CSRs in their review, or had received data but decided not to include it, or were still awaiting for data)
‘Considered’ using but not requested access to CSRs
‘Never considered’ the use of regulatory documentation such as CSRs
Respondents who had previously ‘requested’ CSRs or other regulatory documents were asked to explain their reasons for seeking and for using (or not using) the data, the source and type of documents obtained and how the data were used in the review. They were also asked to describe any difficulties in using provided documents and data, along with their views on the overall importance of seeking data from these sources. Respondents that had only ‘considered’ seeking regulatory documents were asked about what sources they had considered utilising and why they had decided against it; and whether they thought this decision could have impacted on the outcome of their review. Other domains of interest captured were the respondents’ views on the barriers to using data from regulatory documents and what could be done to promote and support the use of such data in future systematic reviews. Those who had never considered the use of regulatory documentation were asked for their views on their potential value and whether they might be encouraged to consider these data in the future.
Important criteria when considering data from clinical study reports and/or other regulatory data
Description of criteria
Monetary cost of the intervention on the healthcare budget (i.e. considering both the price of a course and the number of people in the population that are being—or will be—treated)
Burden of disease of the indication this product is meant to treat/prevent
How many people are using or likely to use this product?
Product new to the market?
Product from a new drug class or has a new mechanism of action
Has important interactions with other drugs (e.g. drug-drug interactions)
High proportion of RCTs evaluating this product are industry funded
Prominent claims of safety and/or efficacy advantage of this product over currently available treatments
High degree of media attention surrounding this product
High proportion of trials of this product are unpublished
Post-marketing surveillance has identified safety concerns?
Important or standard outcome measures (also known as ‘endpoints’) have not been published
Concerns regarding a lack of published data on potential harms of the product
Marketing authorisation based on surrogate outcomes (rather than clinical outcomes)
When protocol(s) are publicly available
When statistical analysis plan(s) publicly available
Known errors or concerns about trial publications of this product
Important discrepancies between the journal publication and the trial registry entry?
Descriptive statistics were used to express quantitative responses including number(s), frequencies and percentages. Verbatim responses were discussed within the team and then tabulated. Responses were generally short and wide-ranging and coded into categories by two research team members (AH, KCD).
A total of 160 respondents completed the first (Cochrane and non-Cochrane) surveys (Fig. 1). Most respondents (93%) were either a Cochrane review author or an editor, 70% worked in academia, 40% were clinicians and 15% were involved in methods research. Of the 160 respondents, 20 (13%) had previously requested or used regulatory data in their review (13 in a Cochrane review and 7 in a non-Cochrane review), 7 (4%) had considered but not used regulatory data and 133 (83%) had never considered using regulatory data.
In the follow-up survey, all 20 respondents who had requested or used regulatory data in a systematic review explained the rationale for making the request, 19 (95%) provided information on where data were requested from and 14 (70%) expressed an opinion about the type of barriers involved. All 7 respondents who considered using regulatory data but did not go on to seek it explained the rationale for doing so, and 5 (71%) gave the primary source of the data under consideration and all responded about potential barriers. For the 133 respondents that had never considered using regulatory data, 91 (68%) indicated that they were familiar with the regulatory process and types of documents produced, 39 (29%) were aware of where they might be able to access regulatory documents such as CSRs and 61 (46%) believed that there are barriers to accessing and using data from these sources.
Rationale for seeking data
For the 20 respondents who had requested or used regulatory data, 15 (75%) believed that regulatory data should be used in systematic reviews and 5 (25%) said that they should be used in some cases (Additional file 3: Table S1 ). Seeking regulatory data was mainly driven by concerns about reporting bias (specifically outcome reporting bias, (n = 11), publication bias (n = 5) and potential for missing data (n = 2) or underreporting of harms (n = 3). The same concerns were raised by respondents who had considered but not sought regulatory data, with 3/7 mentioning outcome reporting bias, and 2/7 underreporting of harms.
For respondents who had never considered using regulatory data, 66/133 (50%) agreed that they should be used in some cases, 43 (32%) said they should definitely be used, 17 (13%) said they should not be used and 7 (5%) said they were unsure about using regulatory data but did not provide any reasons. The reasons given by the 17 respondents who said that regulatory data should not be used were (n = 9) because the interventions explored in their reviews were non-pharmacological, (n = 5) because of lack of guidance on how to find and use these data and (n = 3) because of the time required to obtain the data.
Eight out of 133 (6%) respondents who had never considered using regulatory documents indicated that they had a detailed understanding and 83 (62%) a basic understanding of the regulatory process (Additional file 3: Table S2). However, 42/133 (32%) respondents said that they had no understanding of the regulatory process or the documentation involved. The majority said that they were aware of the ongoing debates and initiatives for improved access to clinical trial data, specifically referring to the AllTrials initiative , the Cochrane review of neuraminidase inhibitors (which was based entirely on regulatory data)  and other publications such as Ben Goldacre’s Bad Science and Bad Pharma. One respondent said that they had been involved in crafting the EMA-led public deliberations regarding the Policy 0070 in 2014, for access to clinical trial data .
Source of evidence
Description of data obtained and how they were used in the systematic reviews
Source of data request(s)
Type of regulatory data/document(s) obtained
Included in meta-analysis
Description of how data were used
‘Summary statistics provided or extracted from the extra documentation were incorporated into meta-analysis’
‘Quantitative data about side effects were included’
‘Data were not used in meta-analyses, but rather in a narrative form instead’
EPARs and Medical Reviews
‘Data was used to describe the number of studies and the number of studied drugs in results of search criteria’
EMA, FDA, Multiple drug companies
FDA and EMA reports, Poster
‘To add data on studies not aware of, and to add outcomes to a published study that were not itemised in the journal publication’
Clinical investigator, EMA, sponsor
No data were obtained
‘Not provided by pharmaceutical sponsor, possibly because study stopped early due to unexpected side effects, and raw data may never have been compiled.’
FDA, Health Canada, NIOSH
Adverse event reports
‘The data did not provide some of the detail we would have liked, such as indication for the drug, dosing etc. We summarized the results in narrative form but did not include in the quantitative analyses of the data we retrieved from published studies’
Clinical investigator, medical director of company
‘Assessed quality of the studies and extracted data for use in forest plots and description’
Clinical Study Data Request, EMA, FDA
Case report forms
‘N/A as data not received’
Clinical investigator, EMA, Pharmaceutical company
Details of trial participants at start of trial (baseline data and info about randomisation)
‘Only data at start of trial was available’
EMA, GSK and FDA
Clinical and Statistical reviews at FDA, CSRs
‘We checked the data for consistency (across multiple published and unpublished sources) and reported in the systematic review the most accurate and conservative estimates. If needed, we contacted authors for confirmation’
‘Data from CSRs & IPD were used in evidence synthesis’
‘We know patient level data exists but we were not given access to it despite trying’
‘Extraction of data from Pfizer Medical Information Report’
CSRs, protocol with appendices
‘We extracted, compared and used the aggregated effect estimates data for predefined outcomes’
Helsinn, Merck and Pfizer
‘Where possible incorporated it as more likely to be the correct data than what was published’
FDA medical and statistical reviews
‘Performed data extraction from these sources. Compared with data from published sources’
CSRs, FDA reports and IPD
‘Data was used in place of publication’
‘Data were used in network meta-analyses’
Bristol-Myers Squibb, Genentech, Schering-Plough
‘In narrative synthesis. However, some of the data/text needed to be removed before the final technology assessment report is published under the confidentiality agreement’.
Of the 133 respondents who had never considered accessing regulatory data, 117 (88%) said they were not aware (or were unsure) of where to access such material (Additional file 3: Table S3). Sixteen (12%) respondents said that they were aware of at least one possible source of regulatory data. The EMA and FDA were the two sources most commonly noted, but other regulatory agencies mentioned were The Health Products Regulatory Authority of Ireland, Pharmaceuticals and Medical Devices Agency of Japan and Therapeutic Good Administration Department of Health Australia. Other sources mentioned but not considered to be specific to regulatory data included the trial registries (ClinicalTrials.gov and the ISRCTN register). The clinical study data request sharing platform was considered by only one respondent. Pharmaceutical companies and ethics committees were also mentioned as other sources of data.
Barriers when seeking regulatory data for use in a Cochrane review
Requested/used regulatory data
Considered regulatory data
Not considered regulatory data
Total no. of responses: n (% of total responses)
Are there any barriers to using regulatory data?
n = 20
n = 7
n = 133
What were these barriers?
n = 14
n = 5
n = 60
Restricted and limited sharing of data
Lack of experience (inc. statistical)
Identifying/searching for trials
Quality of data
Limited access for peer reviewers*
Criteria considered important for using regulatory data
Between 9 (64%) and 13 (93%) respondents considered ‘marketing authorisation based on surrogate outcomes’, ‘safety or efficacy advantage over current treatments’, ‘product new to the market or from a new drug class’, ‘important drug-drug interactions’ and ‘monetary cost of the intervention’ to be important.
The two criteria deemed less important (by 6 (43%) to 9 (64%) of authors) included the public availability of ‘statistical analysis plans’ and of ‘protocols’.
Respondents expressed divided opinion about whether ‘number of people using the product’, ‘high degree of media attention surrounding the drug’ and ‘burden of disease’ were important ‘triggers’.
Respondents suggested as other ‘triggers’ for seeking and using of regulatory data: ‘the lack of clarity on published trials’ and ‘when a small number of trials are available’. It was also noted that ‘regulatory data were irrelevant for non-pharmacological intervention reviews (e.g. surgical techniques, psychological interventions and psychical therapy) and were therefore unable to use CSRs.’
Summary of findings
In this survey, only 27/160 (17%) systematic review authors had used, requested or considered using regulatory data in their review and 133/160 (83%) had never considered using such data. Respondents who had requested regulatory documents had mainly sought these from the EMA and the FDA. Other requests were made to individual pharmaceutical companies, but only one was made to a data sharing platform. Respondents also described seeking data from the clinical investigators or authors of published trials in their responses, although these are clearly not usual sources of regulatory documents, which may indicate unfamiliarity with and a misunderstanding of the question posed as being about any ‘unpublished data’ rather than specifically being focused on regulatory documents.
Clinical study reports were acquired by 12/20 (60%) of the respondents requesting data, but other regulatory documents including Medical and Statistical Reviews from the FDA (5/20 (25%)), European Public Assessment Reports (EPARs) (2/20 (10%)), and protocols, case report forms and post-marketing adverse reaction reports were also obtained. For the respondents who obtained CSRs, 9/12 (75%) had used the data in their review, in order to include unpublished trials in their meta-analyses and to supplement published data. Two of the respondents were still waiting for the data, one respondent noted that the pharmaceutical company could not provide the data because the study for which the request was made was stopped early due to reports of unexpected side-effects and another respondent reported that only baseline data were provided.
At least two thirds of the respondents who requested or considered utilising regulatory data reported a number of barriers to inclusion of such data in Cochrane reviews including restrictions on accessing trial data, the excessive time involved when waiting for data to be released and the resources, costs and effort required when incorporating the data in a review.
The criteria considered to be most important in triggering decisions to seek regulatory data include situations where there are discrepancies between a study publication and its corresponding trial registry entry, known errors or concerns about publications including a lack of data on harms and where important outcome measures (‘endpoints’) are not reported. Safety concerns identified in post-marketing surveillance, situations where a high proportion of trials are unpublished and/or industry funded, and cases where marketing authorization was based on surrogate outcomes were also considered important indicators of when it would be valuable to seek regulatory data. The lack of availability of the trial protocol and statistical analysis plan and media attention about the drug were considered to be ‘less important’. The cost of the intervention, disease burden, population size and characteristics of the intervention (new to market, interactions with other drugs) were of more mixed opinion of importance amongst authors.
Comparison with other research
A previous study  exploring the experiences of Cochrane review authors when searching for, gaining access to and using unpublished data found that a large proportion of Cochrane review authors had searched for unpublished data. Over half (913/1656 (55.1%)) of those who searched for unpublished data were successful in finding it, and over 81% (651/794) who sought these data went on to use them in their review. In that study, most of the unpublished data were obtained from ‘trialists or investigators’. Of 794 author requests in their study, 403/794 (51%) sought summary data (e.g. mean, standard deviation, sample size), 226/794 (29%) missing outcomes (e.g. quality of life), 163/794 (21%) individual participant data (IPD), 96/794 (12%) results of alternative analysis (e.g. intention to treat), 67/794 (8%) data on harms and 45/794 (6%) CSRs. Data from manufacturers were less frequently used in these reviews. One of the concerns outlined by the authors was that searching for unpublished data was time consuming, which aligns with the opinions expressed by respondents in our survey. Despite the perceived importance of CSRs in providing information about adverse effects, a recent study found that of 348 systematic reviews on adverse effects published in 2014, not one of the reviews had stated that they searched for or included CSRs .
Another study  provided in-depth descriptions of some of the experiences of researchers carrying out systematic reviews when searching for and gaining access to unpublished data. That work aimed to provide guidance on best practices for identifying, obtaining and using unpublished data from a variety of sources, but did not consider regulatory documents. The results suggested that authors differed in their understanding of what was meant by unpublished data, including specific outcomes and methodological details. They also reported that data requests were often seen as time consuming and that including such data was considered to be labour-intensive. There was agreement, however, by the majority of authors that searching for and considering unpublished data in systematic reviews was important for helping to improve the general public health.
Accessing regulatory data
Based on interactions with the community, it seems that many researchers undertaking systematic reviews are still unaware of the various data sharing platforms that provide access to regulatory trial documents and datasets. This may be partly because such data sharing platforms are relatively new and evolving. Furthermore, the limited formal guidance available to systematic review authors explaining how to identify and access regulatory data might also explain why they are rarely used or considered. For example, as noted above, the current version of the Cochrane Handbook does not currently discuss regulatory documents and the data that these might contain, where to find these data, or how to include them . In our study, participants were asked what could be done to promote and support greater use of CSRs and other regulatory documents. Most agreed that there is a need for greater understanding about these documents and for guidance on how to search for and access such data. Some mentioned the need for statistical guidance on how to include the data in evidence synthesis, even though the type of (aggregate) data that these documents contain are largely no different to the type of data presented in journal articles and do not need to be handled and analysed differently (data from patient narratives or case report forms included within a CSR will need different handling). There were also concerns about how to interpret highly statistical content within the documents, e.g. efficacy and safety listings data which may require statistical/software expertise to help extract and organise the data.
Limitations of study
As most of the survey questions captured free-text response(s), the replies were varied and some were unclear or lacked enough detail to understand fully. However, responses were discussed by two team members (AH and KCD) who agreed upon an appropriate classification of response.
Although this survey concerned regulatory documents, and in particular CSRs, it was apparent even from the relatively low numbers who responded that some may have misunderstood some of the questions posed. The term ‘regulatory data’ was not defined in the survey, as we were interested in learning how respondents interpreted this and this may explain why some authors who had used other types of data (e.g. IPD or other summary data obtained from trial authors) responded in the survey. In a number of the responses, authors had made multiple data requests for regulatory data and other data, but it was not always clear in their responses which data or document sources were actually used in their systematic review or meta-analysis. The review references provided, or obtained by contacting the authors, helped to resolve some of these uncertainties about how the data were used in reviews.
The initial survey of Cochrane authors was advertised twice in the Cochrane methods digest, which is circulated by email to the whole Cochrane community. We do not know how many people actually received and opened the email, and consequently how many people read the invitation to opt into the survey. This might explain the low response rate (2.2% calculated using the known figure of 7273 registered Cochrane reviews over the last 2 years as the denominator) compared with that achieved in the survey by Schroll et al.  (37%) which sent invitations to participate directly to review authors. The Schroll survey may also have achieved higher numbers of responses because most Cochrane authors understand the concept of using ‘unpublished data’ whereas fewer may have been familiar with the idea of using regulatory documentation. Furthermore, the use of unpublished data is relevant to all systematic reviews, whilst regulatory documents apply only to reviews of interventions that have made, or intended to make, an application for market authorisation. As respondents in our survey may have been more likely to have a greater understanding of the regulatory process and documents produced, than authors who did not participate in our survey, our sample might not be representative of all authors of systematic reviews. The survey cannot be used, and indeed was not intended, to draw any conclusions about the proportion of reviewers accessing CSRs. Rather it aimed to gain insight into the level of familiarity with regulatory sources of data, particularly amongst Cochrane authors, and to get some indication of the potential level of ‘buy in’ to future encouragement to use these sources in Cochrane Reviews, and what support may be needed to facilitate this.
The results from this survey show that data from CSRs and other regulatory documents are being used in a small number of Cochrane Reviews. The survey revealed that the vast majority of respondents thought that accessing and using CSRs and other regulatory documents in systematic reviews was important, suggesting that the Cochrane community may be ready and willing to engage with this source of evidence. The time taken and resource needed to request, receive and use the data was cited as a major barrier, as was the lack of formal guidance on access to and use of documents produced for regulatory purposes. There is a pressing need to develop materials to help review authors identify questions and topics where using regulatory data is likely to matter most and which should therefore adopt and invest in such an approach, and to help them navigate regulatory documents and incorporate data from them in Cochrane and other systematic reviews.
We thank CRD staff members Ruth Walker, Hollie Melton and Matthew Walton for helping with the pilot testing of surveys; Toby Lasserson, senior editor, Editorial and methods department, Cochrane central executive, for providing a denominator for the number of active Cochrane reviews over the last 2 years.
This study was funded by the Cochrane Methods Innovation Fund: http://methods.cochrane.org/methods-innovation-fund-2. The funder had no influence on the study design, interpretation of data or the decision to publish the results.
Availability of data and materials
Anonymised datasets of survey responses are available on request from the corresponding author at firstname.lastname@example.org.
The study protocol was drafted by AH and LS, and other members of this Cochrane Methods Innovation Funded (MIF) project (CL, SG, MJ, PD, CH, TJ and IB) contributed. AH created the online survey which was pilot tested by all MIF members; LS advertised the survey via the Cochrane and Systematic Reviews journal homepages; AH tabulated the data; and KCD checked for consistency. The data were analysed by AH, and AH and LS drafted the manuscript. AH is the guarantor and takes responsibility for the integrity of the data and the accuracy of the data analysis. All other MIF authors CL, SG, MJ, PD, CH, TJ and IB have edited and approved the final manuscript. Referencing was organised by SG and CL. All authors read and approved the final manuscript.
Ethics approval and consent to participate
This study was granted ethical approval by the University of York Health Sciences Research Ethics and Integrity Governance Committee on 16 May 2016.
All authors have completed the ICMJE uniform disclosure.
PD received personal fees from European Respiratory Society, grants from UK NIHR, grants from American Association of Colleges of Pharmacy, grants from Cochrane Methods Innovation Fund outside this work; and is also an associate editor of The BMJ and an unpaid member of the IMEDS steering committee at the Reagan-Udall Foundation for the FDA, which focuses on drug safety research.
TJ was a recipient of a UK National Institute for Health Research grant for a Cochrane review of neuraminidase inhibitors for influenza. Receives royalties from his books published by Il Pensiero Scientifico Editore, Rome and Blackwells. Is occasionally interviewed by market research companies about phase I or II pharmaceutical products. In 2011-13, TJ acted as an expert witness in litigation related to the antiviral oseltamivir, in two litigation cases on potential vaccine-related damage and in a labour case on influenza vaccines in healthcare workers in Canada. He also acted as a consultant for Roche (1997-99), GSK (2001-2), Sanofi-Synthelabo (2003), and IMS Health (2013). TJ is a cosignatory of the Nordic Cochrane Centre Complaint to the European Medicines Agency (EMA) over maladministration at the EMA in relation to the investigation of alleged harms of HPV vaccines and consequent complaints to the European Ombudsman. TJ is co-holder of a John and Laura Arnold Foundation grant for development of a RIAT support centre (2017-2020) and Jean Monnet Network Grant, 2017-2020 for The Jean Monnet Health Law and Policy Network. The authors declare that they have no competing interests except for PD and TJ.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- McGauran N, Wieseler B, Kreis J, Schüler Y-B, Kölsch H, Kaiser T. Reporting bias in medical research—a narrative review. Trials. 2010;11(1):1–15.View ArticleGoogle Scholar
- Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, Forbes A. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database Syst Rev. 2014;10:MR000035.Google Scholar
- Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3:e3081.View ArticlePubMedPubMed CentralGoogle Scholar
- Dwan K, Gamble C, Williamson PR, Kirkham JJ, for the Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS One. 2013;8(7):e66844.View ArticlePubMedPubMed CentralGoogle Scholar
- Eyding D, Lelgemann M, Grouven U, Harter M, Kromp M, Kaiser T, Kerekes MF, Gerken M, Wieseler B. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010;341:c4737.View ArticlePubMedPubMed CentralGoogle Scholar
- International Conference on Harmonisation of Technical requirements for registration of Pharmaceuticals for Human use (ICH): Structure and Content of Clincial Study Reports E3. [http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E3/E3_Guideline.pdf].
- Doshi P, Jefferson T. Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports. BMJ Open. 2013;3(2):e002496.View ArticlePubMedPubMed CentralGoogle Scholar
- Le Noury J, Nardo JM, Healy D, Jureidini J, Raven M, Tufanaru C, Abi-Jaoude E. Restoring study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ. 2015;351:h4320.View ArticlePubMedPubMed CentralGoogle Scholar
- Jefferson T, Jones M, Doshi P, Del Mar C. Neuraminidase inhibitors for preventing and treating influenza in healthy adults: systematic review and meta-analysis. BMJ. 2009;339:b5106.View ArticlePubMedPubMed CentralGoogle Scholar
- Wieseler B, Wolfram N, McGauran N, Kerekes MF, Vervölgyi V, Kohlepp P, Kamphuis M, Grouven U. Completeness of reporting of patient-relevant clinical trial outcomes: comparison of unpublished clinical study reports with publicly available data. PLoS Med. 2013;10(10):e1001526.View ArticlePubMedPubMed CentralGoogle Scholar
- Schroll JB, Bero L, Gotzsche PC. Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ. 2013;346:f2231.View ArticlePubMedPubMed CentralGoogle Scholar
- Hodkinson A, Gamble C, Smith CT. Reporting of harms outcomes: a comparison of journal publications with unpublished clinical study reports of orlistat trials. Trials. 2016;17(1):1–11.View ArticleGoogle Scholar
- Golder S, Loke YK, Wright K, Norman G. Reporting of adverse events in published and unpublished studies of health care interventions: a systematic review. PLoS Med. 2016;13(9):e1002127.View ArticlePubMedPubMed CentralGoogle Scholar
- Gotzsche PC, Jorgensen AW. Opening up data at the European Medicines Agency. BMJ. 2011;342:d2686.View ArticlePubMedGoogle Scholar
- Doshi P, Jefferson T. Open data 5 years on: a case series of 12 freedom of information requests for regulatory data to the European Medicines Agency. Trials. 2016;17(1):78.View ArticlePubMedPubMed CentralGoogle Scholar
- Wieseler B, McGauran N, Kerekes MF, Kaiser T. Access to regulatory data from the European Medicines Agency: the times they are a-changing. Syst Rev. 2012;1(1):50.View ArticlePubMedPubMed CentralGoogle Scholar
- European Medicines Agency (EMA) Policy on publication of clinical data for medicinal products for human use. Policy/0070. 2 October 2014 EMA/240810/2013. [http://www.ema.europa.eu/docs/en_GB/document_library/Other/2014/10/WC500174796.pdf].
- The YODA Project: Forging a unified scientific community. [http://yoda.yale.edu/].
- ClinicalStudyDataRequest.com. [https://clinicalstudydatarequest.com/].
- Duke Clinical Research Institute: From thought leadership to clinical practice. [https://www.dcri.org/].
- +AllTrials. All Trials Registered | All Results Reported. [http://www.alltrials.net/].
- OpenTrials. All the data on all the trials, linked. [https://opentrials.net/].
- Goldacre B, Lane S, Mahtani KR, Heneghan C, Onakpoya I, Bushfield I, Smeeth L. Pharmaceutical companies’ policies on access to trial data, results, and methods: audit study. BMJ. 2017;358:j3334.View ArticlePubMedGoogle Scholar
- Doshi P. From promises to policies: is big pharma delivering on transparency? BMJ. 2014;348:g1615.View ArticlePubMedGoogle Scholar
- Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality. January 2014. [https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-methods-guide_overview.pdf].
- Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: JPT H, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011): The Cochrane Collaboration; 2011. http://training.cochrane.org/handbook.
- Golder S, Loke YK, Wright K, Sterrantino C. Most systematic reviews of adverse effects did not include unpublished data. J Clin Epidemiol. 2016;77:125–33.View ArticlePubMedGoogle Scholar
- Qualtrics. Software to manage the entire customer experience—from surveys, to insights, to action. [https://www.qualtrics.com/].
- The National Institute for Occupational Safety and Health (NIOSH). [https://www.cdc.gov/niosh/index.htm].
- Wolfe N, Gøtzsche PC, Bero L. Strategies for obtaining unpublished drug trial data: a qualitative interview study. Syst Rev. 2013;2(1):1–11.View ArticleGoogle Scholar
- Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]: The Cochrane Collaboration; 2011. Available from http://training.cochrane.org/handbook