- Open Access
- Open Peer Review
Identifying additional studies for a systematic review of retention strategies in randomised controlled trials: making contact with trials units and trial methodologists
Systematic Reviewsvolume 6, Article number: 167 (2017)
Search strategies for systematic reviews aim to identify all evidence relevant to the research question posed. Reports of methodological research can be difficult to find leading to biased results in systematic reviews of research methodology. Evidence suggests that contact with investigators can help to identify unpublished research. To identify additional eligible randomised controlled trials (RCTs) for a Cochrane systematic review of strategies to improve retention in RCTs, we conducted a survey of UK clinical trials units (CTUs) and made contact with RCT methodologists.
Key contacts for all UK CTUs were sent a personalised email with a short questionnaire and summary protocol of the Cochrane methodology review. The questionnaire asked whether a RCT evaluating strategies to improve retention embedded in a RCT had ever been conducted by the CTU. Questions about the stage of completion and publication of such RCTs were included. The summary protocol outlined the aims, eligibility criteria, examples of types of retention strategies, and the primary outcome for the systematic review. Personal communication with RCT methodologists and presentations of preliminary results of the review at conferences were also used to identify additional eligible RCTs. We checked the results of our standard searches to see if eligible studies identified through these additional methods were also found using our standard searches.
We identified 14 of the 38 RCTs included in the Cochrane methodology review by contacting trials units and methodologists. Eleven of the 14 RCTs identified by these methods were either published in grey literature, in press or unpublished. Three remaining RCTs were fully published at the time. Six of the RCTs identified were not found through any other searches. The RCTs identified represented data for 6 of 14 RCTs of incentive strategies (52% of randomised participants included in the review), and 6 of 14 RCTs of communication strategies (52% of randomised participants included in the Cochrane review). Data were unavailable for two of the RCTs identified.
Methodological evaluations embedded in RCTs may be unpublished, published in the grey literature or where published, poorly indexed in bibliographic databases. To identify such studies and minimise selection bias in systematic reviews of methodological evaluations, reviewers should consider contacting CTUs and trial methodologists.
Search strategies for systematic reviews are designed to find all of the relevant evidence to answer a specific research question  and to minimise bias . However, capturing all eligible studies for a systematic review through bibliographic database searching alone can be difficult. This is because studies might be poorly indexed, or the search strategy may not include all the necessary terms, or instead because the studies are hidden in grey literature. Moreover, methodological studies embedded within trials (SWATs) or other studies may be even less accessible. Unpublished studies, in particular, can be difficult to identify, and failure to include these and other difficult to find studies in systematic reviews can lead to under or overestimation of effects . While there is evidence to suggest that contacting investigators is a useful way to find published and unpublished eligible studies that are otherwise difficult to find for systematic reviews of interventions [1, 2], we have not found any examination of different ways of identifying unpublished eligible studies.
To identify additional studies for a Cochrane methodology systematic review of strategies to improve retention in randomised controlled trials RCTs , we conducted a survey of UK clinical trial units (CTUs) and communicated with colleagues working in RCT methodology, as well as undertaking standard searches of bibliographic databases, conference abstracts and reference lists. In this paper, we describe the methods and the results of using these additional search methods.
For the survey, we used the UK Clinical Research Collaboration website (UKCRC), http://www.ukcrc-ctu.org.uk/  to identify a key contact for each UK CTU. The contact details for each key contact were entered into a Microsoft Excel spreadsheet. We sent a personalised email to each CTU key contact with a two-page summary protocol of the Cochrane review attached. The summary protocol outlined the purpose of the review, the inclusion criteria for eligible RCTs, examples of different types of retention strategies used in RCTs, and a definition of the primary outcome for the review, i.e. retention of participants in RCTs. We also sent a short questionnaire (Additional file 1: Appendix 1) asking whether a RCT of strategies to improve retention embedded within a RCT had ever been conducted by the CTU. For CTUs that had conducted such RCT/s, further questions were asked about the stage of completion of the RCT, availability of the RCT protocol, and the publication status. A reminder email was sent to CTU contacts who had not responded after 4 weeks. The survey was conducted in April 2010 between the initial bibliographic database searches for the review which were conducted in February 2009 and before the bibliographic database search updates were conducted in May 2012.
In addition to conducting the survey, we discussed the review with colleagues and contacts we knew in the field of RCT methodology. We also presented a poster of the preliminary results of the review at the Society for Clinical Trials 31st International conference . The poster mentioned that we wanted to identify additional RCTs for the review. We also posted a message on the conference notice board directing delegates to visit our poster and asking for information about any known potentially eligible RCTs (Additional file 2: Appendix 2). Preliminary results were also presented at the 1st UK Medical Research Council (MRC) Methodology conference . At both conferences, questionnaires similar in format to those sent to UK CTUs were available for conference delegates to complete.
We checked to see if eligible studies identified through these additional methods were also found through our standard searches and if any had been missed.
We recorded responses to the survey in Microsoft Excel. For potentially eligible RCTs identified, details of the publication status, and if published, details of the author, year of publication, title and journal were recorded. For unpublished potentially eligible RCTs in progress or completed, details of the RCT title and principal investigator were recorded. We recorded the same details for all potentially eligible RCTs identified through personal communication.
Full copies of each published potentially eligible RCT were sourced, screened for eligibility using methods described previously  and subsequently sent to a second reviewer (GR). For each eligible retention RCT, we proceeded with the data extraction also described previously . We emailed the principal investigator linked with any eligible and unpublished RCT to see if they were willing to share data to include in the review.
Sixty-nine per cent (34/49) of UK CTUs responded to the survey; 22 (45%) of which responded to the initial email, and 12 (24%) CTUs to the reminder email. Sixteen potentially eligible studies were identified. These studies were at different stages of progression, i.e. planned, in progress, completed and unpublished, or completed and published. Seven of these RCTs were eligible for inclusion in the systematic review of retention strategies. At the time of the survey, two RCTs were fully published [8, 9], one was in press , two were published in grey literature: one as a PhD thesis chapter , and one as an appendix to a Health Technology Assessment (HTA) review  (Table 1). The results of two RCTs were unpublished at the time (Table 1). Results for one of these RCTs have since been published . Data for one unpublished RCT and one published RCT identified through the survey were not available for the Cochrane systematic review analysis (Table 2). Investigators provided data for five RCTs allowing these RCTs to contribute to the meta-analyses of incentives and communication strategies (Table 2). Investigators also provided information that contributed to a risk of bias assessment on those five RCTs .
Duplicate sources of the RCTs identified by the survey are shown in Table 1. One RCT in press at the time of the survey was subsequently published  and found in our Medline search updates. Two eligible published RCTs [8, 9] were found in the reference lists of RCTs included in the review  and relevant literature [8, 9]. The two RCTs published in the grey literature [11, 12] and both unpublished RCTs (by Mitchell and Maclennan) were not identified through any other source.
RCTs identified through personal communication
Seven eligible RCTs (from three publications) were identified in the first instance through personal communication [14,15,16] (two Bailey unpublished reports). One RCT was identified through contact with colleagues in the Hubs for Trials Methodology Research , https://www.methodologyhubs.mrc.ac.uk/about/hubs/ . The other six RCTs were identified through contact with colleagues at CTUs locally either before the survey was conducted  or afterward (Bailey, unpublished, ). The contribution of these trials to the meta-analyses in the systematic review is shown in Table 2. Two of the RCT publications identified through personal communication reported two RCTs each [15, 16]. All of the published eligible RCTs identified through personal communication were found in our Medline search updates [14,15,16]. One of the RCTs was also identified by the survey of CTUs , and another published RCT was identified through the Cochrane methodology register . The two unpublished RCTs by Bailey were not identified through any other source.
The additional RCTs identified by the survey and through personal communication with methodologists represented data for 6 of 14 RCTs of incentive strategies (52% of the randomised participants for incentive strategies included in the Cochrane review) and 6 of 14 RCTs of communication strategies (52% of randomised participants for communication strategies included in the review) (Table 2) .
The survey of CTUs and personal communication with colleagues in the field of RCT methodology helped us identify 14 of the 38 RCTs included in the Cochrane systematic review of strategies to improve retention in RCTs . Eleven of the 14 RCTs identified were either published in grey literature, in press, or unpublished at the time. Three of the RCTs identified were published. Six of the 14 RCTs identified were not identified through any other source. We found that principal investigators were willing to contribute results from unpublished RCTs, which made a major contribution to the meta-analyses of incentive and communication strategies.
To our knowledge, this is the first time a survey of CTUs and personal communication with trial methodologists were combined to identify eligible studies to include in a Cochrane methodology review. The response rate to the survey was high(69%) and may be explained in part by the importance given to retention in RCTs by UK CTUs and by using an individualised approach to the survey with a short questionnaire. We also raised the profile of the review by presenting preliminary results and giving details of the survey at trials methodology conferences, which may explain the principal investigators willingness to share unpublished data. As the survey was conducted between the initial database search and the search updates, we were able to check the reliability of our other search strategies in picking up these RCTs and see a net gain of using these extra search methods. The inclusion of additional RCTs improved the overall quality of the review results.
Only one reminder email was sent to the CTUs surveyed. More reminders may have improved the number of responses. It is therefore possible that more unreported retention RCTs could have been identified at the time were more reminders sent. As the survey was conducted in the UK, the results do not include RCTs in progress or unpublished from other regions. Although presentations of the preliminary results of the review at national and international conferences helped to publicise and draw attention to the review inclusion criteria, no new eligible retention RCTs were identified through these means. We checked the results of our initial database searches and found that two of the eligible RCTs identified through the survey were not identified through the initial database searches [8, 9]. This may be explained by a mismatch between the search terms used for our initial database searches, ambiguity around definitions of retention and attrition in RCTs, and indexing terms used to label subjects by bibliographic databases when studies were indexed.
Implications of the results
Personal communication has been used before in methodology reviews to identify unpublished data , and the evidence suggests that email is the best method to use . The UKCRC  website was a useful resource for identifying key contact details for each CTU. We found that conducting the survey by email was efficient; nevertheless, systematic reviewers planning to include such a survey as part of their search strategy may wish to consider the associated additional time and costs. Since our survey was conducted, the Studies Within a Trial (SWAT) database of evaluations of methods for RCTs has been established . Systematic reviewers could also consider searching this resource to identify embedded methodology RCTs in progress for future systematic reviews of research methodology.
It is clear from the results of the survey that RCTs evaluating strategies to improve retention in RCTs remain unpublished or are published in the grey literature. A possible explanation for this is the priority given to publishing the results of host RCTs. Moreover, as these RCTs of retention strategies are embedded within RCTs of health interventions, they have less prominence within a trials report and are likely to be more difficult to publish as stand-alone pieces of research. Thus, clear guidance on the reporting of methodology RCTs embedded in RCTs is needed, which in turn might ensure more accurate indexing in bibliographic databases.
Published eligible RCTs identified through the survey and by personal communication were mostly published since 2005 in journals focused on research methodology, e.g. BMC Medical Research Methodology , Clinical Trials , Trials , BMC Trials , Journal of Medical Internet Research , and Journal of Clinical Epidemiology  (Table 1). This reflects the relatively recent emergence of trial and other research methodology as distinct areas for research. Therefore, hand searching of relevant research methodology journals that are not included in major bibliographic databases would be useful for methodology reviews such as ours. The other RCTs identified for the systematic review were published in journals that reflected the focus of the host trial within which the retention trial was embedded. Some were published in general medical journals, e.g. the British Medical Journal, or in more specific clinical journals such as, Clinical Oncology, Journal of Health Psychology, Child Maltreatment and Gerontologist, reflecting the spectrum of host trials that the retention trials were embedded within. Thus, search strategies to identify SWATs for methodology reviews will need to remain broad and continue to make use of the major medical bibliographic databases.
Methodological evaluations, particularly those embedded in RCTs or other clinical studies, may be unpublished, published in the grey literature or where published, poorly indexed in bibliographic databases. To identify such studies and minimise selection bias in systematic reviews of methodological evaluations, reviewers should consider contacting CTUs and trial methodologists.
Clinical Trials Unit
Health Technology Assessment
Medical Research Council
Randomised Controlled Trial
United Kingdom Clinical Research Collaboration
Lefebvre C, Manheimer E, Glanville J. Searching for Studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. England: Wiley; 2008. Chapter 6 p. 95–150.
Young T, Hopewell S. Methods for obtaining unpublished data. Cochrane Database of Syst Rev. 2011(11). 10.1002/14651858.MR000027.pub2.
Sterne, J. Egger, M. Moher, D. Addressing reporting bias. In: Higgins JPT, Green S, editors. Cochrane Handbook for systematic reviews of Interventions. England: Wiley; 2008. Chapter 10 p. 297–333.
Brueton, VC. Tierney, J. Stenning, S. Harding, S. Meredith, S. Nazareth, I. Rait, G. Strategies to improve retention in randomised trials. Cochrane Database Syst Rev 2013(12):MR000032.
The UK Clinical Research Collaboration website (UKCRC). http://www.ukcrc-ctu.org.uk/. Accessed 3 Aug 2017.
Brueton V. Rait, G. Tierney, J. Stenning, S. Meredith, S. Darbyshire, J, Harding, S. Nazareth, I. Strategies to reduce attrition in randomised trials: a systematic review. Clinical Trials. 2010;7:492.
Brueton V, Tierney J, Stenning S, Nazareth I, Meredith S, Harding S, et al. Strategies to reduce attrition in randomised trials. Trials. 2011;12(1):A128.
Brown AP, Lawrie HE, Kennedy AD, Webb JA, Torgerson DJ, Grant AM. Cost effectiveness of a prize draw on response to a postal questionnaire: results of a randomised trial among orthopaedic outpatients in Edinburgh. J Epidemiol Community Health. 1997;51(4):463–4.
Cockayne S, Torgerson D. A randomised controlled trial to assess the effectiveness of offering study results as an incentive to increase response rates to postal questionnaires [ISRCTN26118436]. BMC Med Res Methodol. 2005;5(1):34.
Ashby R, Turner G, Cross B, Mitchell N, Torgerson D. A randomized trial of electronic reminders showed a reduction in the time to respond to postal questionnaires. J Clin Epidemiol. 2011;64(2):208–12.
Nakash RA. A study of response and non-response to postal questionnaire follow-up in clinical trials: Chapter 6: A randomised controlled trial of a method of improving response to postal questionnaire followup in a clinical trial. Coventry: University of Warwick, Warwick Medical School; 2007.
Marson A, Appleton R, Baker G, Chadwick D, Doughty J. A randomised controlled trial examining the longer-term outcomes of standard versus new antiepileptic drugs. The SANAD trial. Health Technol Assess. 2007;11(37):145.
MacLennan G, McDonald A, McPherson G, Treweek S, Avenell A. Advance telephone calls ahead of reminder questionnaires increase response rate in non-responders compared to questionnaire reminders only: the RECORD phone trial. Trials. 2014;15(1):1–5.
Gates S, Williams M, Withers E, Williamson E, Mt-Isa S, Lamb S. Does a monetary incentive improve the response to a postal questionnaire in a randomised controlled trial? The MINT incentive study. Trials. 2009;10(1):44.
Khadjesari Z, Murray E, Kalaitzaki E, White IR, McCambridge J, Thompson SG, Wallace P, Godfrey C. Impact and Costs of Incentives to Reduce Attrition in Online Trials: Two Randomized Controlled Trials. J Med Internet Res. 2011;13:1-15.
Severi E, Free C, Knight R, Robertson S, Edwards P, Hoile E. Two controlled trials to increase participant retention in a randomized controlled trial of mobile phone-based smoking cessation support in the United Kingdom. Clinical Trials. 2011;8(5):654–60.
MRC Hubs for Trials Methodology Research. https://www.methodologyhubs.mrc.ac.uk/about/hubs/. Accessed 3 Aug 2017.
Education section – Studies Within A Trial (SWAT). Journal of Evidence-Based Medicine. 2012;5(1):44–5.
Edwards PJ, Roberts IG, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, et al. Methods to increase response rates to postal and electronic questionnaires. Cochrane Database of Systematic Reviews 2009, Issue 3 Art No : MR000008. 2009(3).
Nakash R, Hutton J, Jorstad-Stein E, Gates S, Lamb S. Maximising response to postal questionnaires—a systematic review of randomised trials in health research. BMC Med Res Methodol. 2006;6(1):5.
We would like to thank all UK CTUs that responded to the survey and all principal investigators who shared unpublished data to include in the systematic review of strategies to improve retention in RCTs.
The Cochrane systematic review was funded by the Medical Research Council Population Health Sciences Research Network grant number PHSRN 30.
Availability of data and materials
The data used during the current study is available from the corresponding author on reasonable request.
VB drafted the paper with comments from JT, SS and GR. The summary protocol, reply form and email to CTU contacts were drafted initially by VB with feedback on the content from all authors. VB conducted the survey, managed and analysed the survey data and interpreted the results. All authors read and approved the final manuscript.
Ethics approval and consent to participate
The survey of UK CTUs focused on identifying additional eligible studies to include in a Cochrane systematic review and was conducted in full knowledge of the senior management at the MRC CTU. Consent to participate in the survey was considered given when the CTU key contact replied to the initial email. The Cochrane review contributed to a PhD thesis. Ethics approval for a qualitative study associated with the PhD thesis was sought from the University College London Ethics Committee UCL 2342/002.
Consent for publication
No individual person’s data was used for this research.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
- Systematic review methods
- Unpublished data
- Searching for methodological research
- Personal communication
- Methodology review