This SR identified and synthesised results from 15 meta-research studies representing 1724 original studies examining if existing studies in clinical research use SRs to place their results in the contexts of earlier studies. On average, approximately one third of the original studies placed their results in context in the ‘Discussion’ section with a mean percentage of 36.8% and ranging from 9.1 to 48.1%. Less than one fifth referred to a relevant SR (the term relevant was defined by the authors of the included studies). Only one original study [31] integrated their results with existing studies quantitatively with an MA, while four updated an SR [20, 29,30,31] (Hoderlein et al. [20] Cohort 2). The results display great variation between original studies and during the period of assessment. Even the five meta-research studies with fully identical inclusion criteria but conducted years apart presented fluctuating results [18, 27,28,29,30]. Overall, less than half of the original studies contextualised their results with existing evidence, and only a small fraction did so quantitively.
Possible positive progress might appear concerning the prevalence of meta-research studies in the field, as 8 of 15 meta-research studies were published the last 5 years of the 24-yearlong study period [19,20,21,22,23,24,25,26]. Yet, the individual results did not show the progress of more clinical researchers using an evidence-based approach over time, as the percentage of original studies placing the results in context did not differ from the earlier meta-research studies. Additionally, three of the recent meta-research studies [19, 21, 24] did not present any data on SRs in the ‘Discussion’ section. Possible positive progress might also be noticeable in the number of included original studies in each meta-research study, as those including less than 100 original studies were published from 1998 to 2017 [18, 20, 23, 27,28,29,30,31,32] (Hoderlein et al. [20] Cohort 1), compared to those including more than 100 original studies, which were published from 2015 to 2021 [19,20,21,22, 24,25,26] (Hoderlein et al. [20] Cohort 2). However, this was not succeeded by an increased rate of contextualising, as the rate of citing or updating an SR did not differ between the two groups of meta-research studies. In the same way, the meta-research studies did not display any difference depending on whether the area of interest for the original studies was focused on a specific speciality [19,20,21,22,23, 25, 26, 32] or not [18, 24, 27,28,29,30,31]. Thus, practising EBR in the way of contextualising new clinical results has not improved over time and is not conditional on the number of original studies in the meta-research studies nor the area of interest.
Another noticeable feature of our results is that all meta-research studies were based on original studies published in high-ranking journals, except Hoderlein et al. [20], which was based on studies from a specific physiotherapy database. Some journals, for example, The Lancet [10,11,12], require authors of new studies to place their results in the context of earlier evidence when publishing. Therefore, our results might present an overestimation of contextualising new results in general, as the requirement of contextualising is not standard for all journals.
Employed definitions of SRs
Our applied definition of the use of SRs to place results in context as individually interpreted by the authors naturally has implications for our results. Meta-research studies used terms such as integrated results [20, 29,30,31], referred to relevant SR [18, 23, 26, 27, 32], stated a comparison, but no further discussion [31], attempted to discuss or explain in relation to other trials [31], cite [19, 21, 22, 24, 25] and summarised some evidence [20]. This broad definition implies that our sample of original studies includes a wide range of studies from those citing an SR to those updating an SR or conducting an MA and is dependent on the authors’ interpretations. The strive for comprehensiveness implies high sensitivity and heterogeneity, and our results must, therefore, be interpreted with caution and as merely representing any original study, at least mentioning existing knowledge in the field of interests, but not necessarily contextualising new evidence with existing evidence. Consequently, our results might reflect an overestimation of the extent of contextualising when finishing new studies, if contextualisation implies relating new results with earlier, similar results and not just mentioning.
It is important to acknowledge that the prevalence of placing new results in the context of earlier results depends on the prevalence of earlier similar studies. One way to control for this might be the authors stating whether earlier similar studies were searched for and located. Eight of the meta-research studies [18, 23, 27,28,29,30,31,32] reported the number of original studies that ‘claimed to be the first’ original study. However, only three meta-research studies [18, 27, 28] assessed whether this claim was true. Our results maybe have to be moderated by the fact that one cannot assume the prevalence of earlier, existing evidence in all clinical areas.
Further implications
Our results show pronounced room for improvement when finishing a new study. When results of new clinical studies are not contextualised with existing knowledge in the field, no building up on the knowledge base in a specific clinical area, and thereby no way of establishing whether a new study adds new knowledge or confirmed what we already know [33]. By doing so, it is not possible to establish whether a research question is solved, or further research is needed. This might lead to inefficient use of research funding within a precise clinical question that might already have been answered [5, 34, 35]. Contextualising new results in the way of practising EBR by updating an SR function to prevent redundant research [36].
Furthermore, when not contextualising new results with earlier, similar results, further studies in the area might, therefore, be based on incomplete knowledge in terms of groundings for both the justification and designing of a new study [5, 37,38,39]. It constitutes a potential bias when only a selective sample of original studies make up the knowledge base in a certain clinical area [13, 36], and it resembles publication bias in terms of not improving the basis on which further studies are based. Practising an EBR approach by automatically updating an SR with results from new studies would improve future clinical studies and potentially increase efficiency in the use of research resources [36]. Although we agree there is an extra workload after finishing an original study when having to update an SR afterwards, which might serve as a barrier to updating [3, 7]. We acknowledge the findings of our study should be validated and placed in the context of other similar studies. However, the comprehensive literature search behind this study, identifying more than 30,000 hits, did not identify a similar study.
Strengths and limitations
This study was based on a comprehensive search and screening process for meta-research studies and was conducted by a large group of experienced researchers in the field of meta-research. This is a substantial strength, but the literature search was also contingent on the possibility of locating relevant meta-research studies, as no MESH terms exist for meta-research, EBR, or similar concepts, which constitute the data in our study. We, therefore, had to be even more meticulous in the second search and used words, phrasing and sentences identified in the first search as key terms in the second search. That left us with a high degree of sensitivity and noise, and we prioritised not missing any relevant studies.
We limited the number of databases in the second search after we tested how many of the already identified relevant studies in the first search could be identified in a MEDLINE and Embase search and whether those not identified in MEDLINE could be identified searching Embase and Scopus. As 47 of 49 tested references were identified in the MEDLINE search and two were identified in Embase and Scopus, we limited the updated search to MEDLINE and Embase.
While the application of a risk of bias checklist created specifically for this study may be seen as a compromise to the risk of bias assessment of the studies, this was a necessary step because no applicable checklist was available. Our checklist was developed based upon other risks of bias tools, including the Cochrane Risk of Bias tool, as well as continuous discussion among six researchers experienced in meta-research, evidence-based medicine, and SR methodology. The reduction in a number of items from 13 to 10 in the first draft of the risk of bias tool, and the omission of reporting quality constitutes the only deviations from the registered protocol and represents an adaptation to this study’s specific aim and setting and displays the profound underlying innovative work behind this study. However, we fully acknowledge that we used a custom-made checklist for these specific meta-research studies, and it needs further elaboration, validation and reliability testing.
Furthermore, the degree of heterogeneity among the studies in the MA calls for elaboration. First, the aims of the meta-research studies varied, and used wording was, for example, analyse whether existing SRs were mentioned [26], make use of previous trial evidence in the reporting [31] and discuss new results in light of available evidence [18, 27,28,29,30]. These different study aims naturally have impacted the way each meta-research study was conducted, including the choice of definition of the use of SRs to place results in context as presented above. Furthermore, the meta-research studies differ by their selection of included original studies. Although all included RCTs or MAs, some were limited to a specific area of interest, for instance, physiotherapy [20], pharmacological treatment [32] and urology [19], whereas others were based on original studies in general medical journals, for instance, Clarke et al. [18, 27,28,29,30], Goudie et al. [31] and Walters et al. [24]. Moreover, one meta-research study was not confined to specific journals but a certain database [20]. In addition, the time of the study period varies from 1998 [18] to 2021 [19]. This timeframe for assessing the degree of contextualisation might be seen as too long—or starting point as too early—as the academic debate around the theme accelerated from about 2005 [12], and the term EBR was introduced in 2011 [38]. Following this line of argument would imply an increase in meta-research studies that place their results in context with earlier, similar studies during the latest year. However, this presumption was not supported by our results.
It is worth noting that 5 of the 15 included meta-research studies had the same first author [18, 27,28,29,30] and another five meta-research studies partly had a common group of authors [19, 21, 22, 24, 25]. These two groups of meta-research studies, therefore, represented a large degree of homogeneity compared to the other group and the other five of the included meta-research studies; thus, their priorities had a relatively higher weight in the study material.
The results of this SR are characterised by a high degree of heterogeneity and should be interpreted cautiously. We recommend subgroup analyses when future, similar meta-research studies are conducted. The underlying cause of heterogeneity is not identifiable but could be due to the broad range of clinical health specialities represented among the meta-research studies and methodological features of the meta-research studies. Given the range of clinical specialities, timeframes and differences in approaches to conducting the meta-research studies, the results of this SR cannot be regarded as representative of all clinical trials in health care.