Since its inception in 2012, SRDR has accumulated a corpus of 152 systematic review projects with publicly available data from more than 15,000 studies. Data from these projects and studies can be accessed by anyone around the world to review, re-use in a new systematic review or related research project, conduct methodologic research, or otherwise use for various purposes. Almost two-thirds of these projects include data in a structured format. A majority of the 152 projects are in clinical fields, focus on interventions or diagnosis, and are funded by government sources. The projects cover various health areas, with mental and behavioral disorders and diseases of the eye and ocular adnexa being the most common.
Comparison with other investigations
There are some interesting differences between the findings of this investigation and others that have examined the characteristics of systematic reviews in health. For example, others have found that about half of systematic reviews have focused on interventions and that about 45% have been funded by government sources [11, 12]. In SRDR, 72% of publicly available reviews focused on interventions and 91% were funded by government sources. These higher proportions in SRDR are largely driven by the large proportion of reviews in it that have been funded by AHRQ (68%) or are Cochrane systematic reviews (15%). AHRQ-funded and Cochrane systematic reviews tend to focus more on interventions than do other systematic reviews. Similarly, another investigation found that the median number of included studies per systematic review was 15 [11], whereas the median number of studies per review in SRDR is 70. This difference is also largely driven by the predominance of AHRQ-funded reviews, which tend to be broader in scope and address more research questions than non-AHRQ-funded systematic reviews. Indeed, in SRDR, the median number of research questions per AHRQ-funded review is approximately eight times that of non-AHRQ-funded reviews.
Our finding that 44% of eligible systematic reviews were registered in PROSPERO is a considerable improvement from the 4% that was reported in a random sample of 300 systematic reviews published in 2014 [11]. However, since AHRQ and Cochrane strongly encourage PROSPERO registration, we would expect reviews in SRDR to be more likely to have registered protocols than other systematic reviews. While the higher percentage is a good sign, we urge all systematic review teams to register their systematic reviews prospectively in PROSPERO. Prospective registration offers many benefits, such as promoting transparency, reducing the potential for bias, and reducing the potential for redundancy [13, 14]. Additionally, in light of the growing numbers of abridged types of evidence syntheses, such as evidence maps/scoping reviews [15], living systematic reviews [16], and rapid reviews/technical briefs [17], we agree with Page et al. [12] that the PROSPERO registry should expand its eligibility criteria to include these other types of reviews.
Potential value of publicly available data from systematic reviews to the global community
To our knowledge, SRDR is one of a kind. It serves as a free, online, data management platform for collaboration among members of a systematic review team [8]. SRDR also offers free, open access to data about primary studies that have been extracted for systematic and other reviews on a range of topics. In this way, SRDR helps advance the open-access movement in science. We agree, however, with current guidance that those re-using shared data should cite the original data source (i.e., the systematic review and the SRDR platform from which the data were obtained) [5, 18]. To facilitate such citing, we provide each publicly available project in SRDR with an associated digital object identifier (DOI) for easy and persistent online identification.
The past decade has witnessed an almost threefold surge in the number of systematic reviews [11]. While some of these systematic reviews have been demonstrated to be redundant [19], conducting new systematic reviews on topics related to existing systematic reviews is often required. For example, an update to an existing systematic review may be needed if it is out of date and/or new studies are available. Other common scenarios that lead to a new systematic review being needed are when the eligibility criteria of an existing related systematic review were too narrow, a new type of intervention or comparator has emerged, or a broader search is needed [20]. In each of these and other related scenarios, a considerable amount of time and resources can be saved by using previously extracted data, where relevant. SRDR can help fulfill this need [8].
Efforts have been underway to begin strides towards a future in which data extraction for systematic reviews might be accurately and efficiently conducted using automation technologies [21,22,23]. In that context, while archival of data would still serve the purpose of transparency, re-use of data might not be of much added value because technology would be able to conduct data extraction inexpensively. While such a “revolution in automation of systematic reviews” is on the horizon, we are not there yet [24]. Until we arrive at such a future, one in which the systematic review and broader research communities are comfortable with the accuracy of automated data extraction, re-use of data extracted by humans can help reduce redundancy and costs.
There appears to be support for sharing systematic review data. A 2014 survey found that 83% of systematic reviewers affiliated with the Cochrane Individual Participant Data (IPD) Review Group supported it [25]. However, we acknowledge that there might be barriers to completely relying on previously extracted data, especially when the previous systematic review team is different from the team undertaking the new systematic review. There may be concerns about whether the set of items extracted from studies in the previous review is adequate and whether the data extracted are accurate. While SRDR does not guarantee the adequacy and accuracy of extracted data, these aspects can be assessed by systematic review teams, such as through examination of data from a random sample of the studies. Alternatively, systematic review teams may choose to use the previously extracted data as the initial extraction in an approach similar to single (de novo) data extraction and verification.
SRDR can serve as a valuable platform for conducting methodologic research. Examples of such research that has already been conducted using SRDR are the Data Abstraction Assistant (DAA) Trial (a randomized controlled trial that compared different data extraction approaches [26,27,28]) and the current study and the six other methodologic projects described in this paper [29,30,31,32,33,34]. SRDR also can serve as a source of data for meta-research (i.e., methodologic and other types of research on research [4]). For example, researchers might analyze the populations, interventions, comparators, outcomes, funding sources, and/or other factors across systematic reviews, either within or across health areas.
Accessing and downloading publicly available data from SRDR is relatively straightforward through the SRDR Published Projects Website (available at https://srdr.ahrq.gov/projects/published). As the SRDR management team, we are happy to help and/or partner with researchers to do so (see author contact information).
Limitations to publicly available data on SRDR
While the SRDR management team encourages and helps systematic review teams to make their data public and manages the website, we do not monitor the accuracy or completeness of the data. Inaccuracies in publicly available data in SRDR occur due to errors in data extraction from reports of primary studies [35]. It is not uncommon for such errors to be corrected in the final versions of reported data (e.g., journal publications), but not in SRDR. Because SRDR is not yet a data analysis platform, systematic review teams might not be vigilant about retrospectively updating data in SRDR to fix errors that might have been detected during data cleaning or analysis after exporting the data outside SRDR to statistical software or other applications. Second, SRDR is an evolving platform that is striving to improve how best to archive systematic review data for easy re-use. For example, while SRDR offers structured data extraction forms and structured data entry, these features are not always fully taken advantage of; more than one-third of the projects in SRDR have simply uploaded data as flat files. We recently developed an improved mechanism through which systematic reviews can import data from flat files into forms in SRDR so that the data can be shared in a structured format [36]. Third, it should be acknowledged that data have been made available for only a quarter of all projects in SRDR. While we, as the SRDR management team, do not require teams of systematic reviewers to make their data available, we encourage them to do so as quickly as possible. We believe that the median time from SRDR project initiation (i.e., data extraction phase) to public availability of data of 4 months is satisfactory. While we do not track the direction, magnitude, or statistical significance of results of systematic reviews in SRDR, a recent study by other investigators has demonstrated that statistical significance of results is not associated with the duration from protocol registration in PROSPERO to journal publication (for non-Cochrane systematic reviews) [37].
It should be noted that, since 2015, AHRQ has required EPCs to make each of their AHRQ-funded review’s data available publicly through SRDR upon completion of the review. For systematic reviewers working on non-AHRQ-funded projects, there is less of an incentive to make data available publicly. We strongly agree with Wolfenden and colleagues that platforms, such as SRDR, that make systematic review data available publicly can help maximize returns on the significant investments that are made in the systematic review enterprise [5]. We urge more systematic review teams to make their data available publicly.
Limitations to this study
The characteristics of the systematic reviews reported in this study are not intended to be representative of all systematic reviews. As discussed, most of the systematic reviews are AHRQ-funded or Cochrane reviews, leading to a predominance of reviews addressing clinical fields, evaluating interventions, and including higher numbers of studies than has been demonstrated in cross-sections of systematic reviews in other investigations.