Review aims and research questions
This study aims to systematically map and review the outcomes and measurement instruments and to report the prevalence of validity evidence for these measurement instruments in RCTs of eLearning interventions for pre-registration medical education.
We hypothesise that there is a shortage of measurement instruments which provides validity evidence that support learning outcomes such as knowledge, skills, attitudes, competencies, learning satisfaction and patient outcomes in RCTs in eLearning pre-registration medical education. The shortage undermines the credibility of eLearning research leading to the unwarranted interpretation of the results.
In particular, the broad research questions that we aim to answer are:
-
1.
What type of outcomes (e.g. knowledge, skills, attitudes and behaviours) and measurement instruments are used in RCTs of eLearning for pre-registration medical education?
-
2.
What proportion of RCTs on eLearning pre-registration medical education presents validity evidence for the measurement instruments used and how is this evidence reported?
In response to the research questions, we will perform a systematic mapping of research evidence including the types of outcomes, outcome measurement instruments and the proportion of RCTs reporting validity evidence for the measurement instruments used. As part of this systematic mapping review, we will develop a database of eLearning outcome domains and the types of measurement instruments used in relation to each respective domain. We will also search the references of the included RCTs for relevant articles that present validity evidence including psychometric properties of the employed measurement instruments and report their findings. However, in-depth analyses of the quality of validity evidence and synthesis of validity evidence will be performed in another systematic review.
Eligibility criteria for considering studies for this review
Types of studies
We will consider the studies eligible for inclusion if they fulfil the following criteria:
-
(i)
RCTs;
-
(ii)
Involve students participating in pre-registration medical education in any geographical setting or educational setting;
-
(iii)
Evaluate any type of blended or full eLearning method, including a range of eLearning modalities, for example, mLearning, massive open online course (MOOC), online learning, offline learning, virtual reality simulations and digital game-based learning;
-
(iv)
Employ a type of control intervention such as traditional learning, no intervention, other types of eLearning and blended learning.
We adopted the meaning of pre-registration or undergraduate medical education from World Health Organisation by which it means “any type of initial study leading to a qualification that (i) is recognized by the relevant governmental or professional bodies of the country where the study was conducted and (ii) enables its holder primary entry into the healthcare workforce” (p. 11) [4]. Studies will be excluded if they focus on traditional and complementary medicine as defined by WHO [18].
Search strategies
The search strategy of this review aims to find both published and unpublished studies. No language restrictions will be applied. The search for eligible studies will involve both electronic sources and non-electronic sources.
For the electronic search, the following databases will be searched from January 1990 until August 2017:
-
○ MEDLINE (Ovid)
-
○ EMBASE (Elsevier)
-
○ Cochrane Central Register of Controlled Trials (CENTRAL) (Wiley)
-
○ PSYCINFO (Ovid)
-
○ Educational Resources Information Center (ERIC) (Ovid)
-
○ Cumulative Index to Nursing and Allied Health Literature (CINAHL) (EBSCO)
-
○ Web of Science Core Collection (Thomson Reuters)
We will use the MEDLINE strategy and keywords presented in Additional file 1. This will be adapted to search the other databases. A librarian will be consulted when we adapt the search criteria from MEDLINE to other databases. The reason for selecting 1990 as the starting year for our search is because prior to this year, the use of the computers was limited to very basic tasks. We will search reference lists of all the studies that are deemed to be eligible for inclusion in our review and relevant systematic reviews. We will also search the International Clinical Trials Registry Platform Search Portal and metaRegister of Controlled Trials to identify unpublished trials.
Search results across databases will be merged using the reference management software EndNote (X7.2.1) or Covidence. Duplicate records of the same report will be removed. Two reviewers will independently examine the titles and abstracts of the records retrieved from the search. The full-text versions of the potentially relevant studies will be retrieved and assessed against the eligibility criteria. Multiple reports of the same study will be linked together, in order to determine if the study is eligible for inclusion. Both the initial screening and the full-text screening will be done independently by two reviewers. Reviewers will correspond with each other to make final decisions on the study eligibility and resolve any disagreements by a discussion with a third reviewer acting as an arbiter if needed. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Flow Diagram will be used to report the selection and inclusion of studies [19].
Data extraction
Two reviewers will independently extract and manage the data for each of the included studies using a structured data recording form. It will include information such as reference of the study, country of the study, the WHO region of the study, name of measurement instrument, description of measurement instrument, types of outcome, assessment category of measurement instruments [13], assessment method of measurement instruments, type of participants, sample size, raters of the instrument, procedure of identifying the raters and training of the raters for the instruments. We will record any sort of validity evidence sources and measurement properties which are reported directly in the articles such as validity, reliability and responsiveness [14, 20]. We will also record any validity evidence which is recorded indirectly, for example, a reference is given to a validation study to a particular instrument. Additional data about outcome measures, instruments and validity evidence will be recorded verbatim if there is a record of validity evidence. If there is more than one outcome measure, relevant details of the second outcome measure will be recorded. The data extraction form will be piloted and amended according to the received feedback. We will contact the study authors for further data in case of missing information. Disagreements between the reviewers will be resolved by discussion. Data will be extracted from all included studies by two reviewers independently. A third reviewer will act as arbiter.
Data analysis and synthesis
Data will be analysed and synthesised as follows. We will:
-
(i)
Ascertain the types of primary and secondary outcome measurement instruments
-
(ii)
Classify and map the data according to types of outcome (e.g. knowledge, skills, attitudes, satisfaction or competencies); intervention (e.g. online versus offline computer-based eLearning); healthcare profession (e.g. doctors or nurses or allied health professions); types of measurement instruments (e.g. multiple choice questionnaires versus structured direct observation with checklists for rating); and discipline (e.g. anatomy or physiology or pathology)
-
(iii)
Determine the proportion of eLearning RCTs employing measurement instruments with adequate validity in relation to the goal of the measurements (“validity evidence”)