Usability evaluation methods employed to assess information visualisations of electronically stored patient data for clinical use: a protocol for a systematic review
Systematic Reviews volume 6, Article number: 148 (2017)
The use of electronic records in healthcare is increasing. To avoid errors, it is essential that the data displays used by these systems are usable: efficient, effective and satisfying. A wide variety of display techniques are used to present clinical data, but the best methods to assess the usability of these techniques have not been determined.
This systematic review will answer the question: What methods are employed to assess the usability of electronic visualisations of patient data for clinical use? The results of this systematic review will then be used to inform best assessment and design practice.
MEDLINE, EMBASE, CINAHL, OpenGrey, and the Cochrane Database of Systematic Reviews will be searched for original studies related to the usability of electronic information visualisations of patient data for clinical use. Reference lists of eligible studies and relevant reviews will be explored to identify further eligible studies.
This systematic review will identify methods used to assess the usability of electronic information visualisations of patient data for clinical use. We will summarise the similarities and differences between the methods found. Our results will inform best practice when developing new user interfaces to display electronic patient data for clinical use.
There is an increasing use of electronic patient record (EPR) systems in healthcare. The UK government has a target for a comprehensive electronic health system in England by 2020 .
Clinicians use EPR systems to make treatment decisions. It is essential that the information presented in these electronic visualisations is usable—efficient, effective and satisfying. If clinical data are not displayed in a usable format, this can compromise care, for example, by leading to delays in diagnosis or treatment.
A wide variety of visual techniques are used to display clinical data. Usability tools are used for their assessment. How and why individual tools have been used has not been studied. Given the heterogeneity of systems, it is unlikely that one assessment tool would allow accurate assessment or comparison of them all.
A systematic review published in 2014 looked at the use of visualisation techniques and evaluated innovative approaches to information visualisation of electronic health record data . Whilst this review acknowledged the need for a usable design (one of the themes identified), it did not look at the tools and techniques to assess the usability of the different information visualisation techniques.
This systematic review aims to identify and systematically review the methods used to assess the usability of information visualisations of patient data for clinical use. The review will for the first time summarise the methods used to assess usability of information visualisations of individual patient data for clinical use. The strengths and weaknesses found for different techniques in different situations will be discussed. This will allow those developing such software to undertake development in light of the assessment methods that they should use to ensure quality software. It will also allow customers to ensure products they require are suitably assessed and meet their needs.
Studies satisfying all the criteria below will be included:
Describes a method of electronic information visualisation used to display patient data AND includes an assessment of the information visualisation
Is related to electronic visualisations either prototype or used in clinical practice
Is related to methods for evaluating usability of the visualisation technique used to display the medical data
Displays individual patient data, not just patient cohort data
Is for clinician’s use only and not for patient access
Is not related to dentistry
Systematic reviews will not be included in the review, but appropriate studies referenced in reviews will be included. Correspondence and short communications will be excluded. Doctoral research will be included. As our systematic review aims to identify which methods have been used and where these were informative, we will not include protocols without usage data.
The study design is a systematic search of the medical literature followed by a narrative synthesis of the results.
Individual patient data display is implemented in any healthcare settings where it is used for clinical use.
There will be no restriction placed on the time frame of the studies.
Studies published from 1996 onwards will be considered.
No language restrictions will be applied.
Literature search strategies will be developed using Medical Subject Headings (MeSH) and text words related to the usability of information visualisation methods for displaying patient medical data for clinical use.
The following databases will be searched: MEDLINE (Ovid), EMBASE (Ovid), CINAHL (HDAS), OpenGrey and the Cochrane Database of Systematic Reviews (Wiley).
Reference lists of eligible studies and relevant reviews will be explored to identify further eligible studies.
A draft of the search strategy was developed by three of the authors (AB, LM and TP, a medical librarian). The proposed search strategy is shown in Additional file 2.
The citations and full text of papers identified from the search will be stored using Mendeley. A data extraction form will be developed in Excel and piloted.
Literature search results will be uploaded to Covidence, an online software program designed to improve the production of systematic reviews.
Two assessors will independently screen title and abstracts of papers returned by the search against the inclusion criteria. If there is uncertainty from the abstract, the full text will be reviewed. Papers selected on title and abstract will be full-text screened for eligibility. Eligible papers will be included in the study.
Disagreements about eligibility will be resolved by a third party. We will record the reasons for excluding the studies.
The study selection process is illustrated by the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flow diagram in Additional file 3.
Data collection process
Data extraction forms will be created using Excel and piloted prior to use. Two reviewers will independently extract data from the full text of eligible papers. Any uncertainties regarding data extraction will be resolved by discussion amongst the authors.
Data items extracted
We will extract the following data items from each publication:
Date of study
Period of data collection
Type of study design
Clinician/users of system demographics
Type of task visualised
Country of study
Information contained in the display, including for example:
Static: patient demographics, admission history, past medical history
Semi static: ward/bed number
Dynamic: laboratory data, vital signs, neurological status, respiratory status, cardiovascular status, assessment/warning scores, drug charts, fluid charts and medical notes of various clinician groups
Visualisation techniques/display categories as described by Starren et al. :
Graph (simple chart/configural chart/graph notation)
Icons (atomic icon/iconic language)
The following display design features will be identified and recorded:
Interaction level: display, input, alerting and messaging capabilities
Some publications may refer to more than one category of visualisation technique or type of information. Multiple data points will be captured where relevant. The pilot phase of the data extraction form may identify other relevant information which will also be collected.
We will extract data on how usability was assessed in the following categories:
Usability assessment technique/tool type
Test/retest performance of tool
Inter-rater performance of tool
Outcome scale generated
Any assessment of the tool’s user-friendliness
Specific data required by the tool
We will also extract data, where available, on whether the tool records the following categories suggested by Kopanitsa et al. :
Efficiency: time to complete tasks (% of tasks fully completed, % of tasks half completed); comparing task completion quality using software/new software compared to without software/previous software
Effectiveness: % of errors; surveys on % of participants’ responses to task completion; comparing task completion quality using software/new software compared to without software/previous software
Satisfaction: % of participants who make positive/negative comments about the system
This list of performance metrics is not exhaustive, and other measures identified in the literature will also be used.
Outcomes and prioritisation
The primary outcome will be to document methods employed to assess the usability of electronic information visualisations of patient data for clinical use.
Secondary outcomes will include a narrative comparison of the identified methods.
Risk of bias of individual studies
The methodological quality of the studies will be assessed using a modified Downs and Black (D&B) checklist . This checklist was designed to provide an evaluation of the quality of both randomised and non-randomised studies of healthcare interventions on the same scale.
The D&B checklist will be modified to create a 20-question checklist by omitting questions 5, 9, 12, 14, 17, 25 and 26. These questions are deemed not appropriate for assessing the methodological quality of the studies which will be evaluated.
We anticipate that the data extracted will be most appropriate for qualitative synthesis. This will be presented as a narrative synthesis. Themes will be identified in the text, and tables will be used to summarise and explain the characteristics/findings of the included studies.
Currently, there are a variety of information visualisation methods used to present electronic medical data for clinical use . However, there is minimal evidence in the literature regarding assessments of the usability of these systems, despite a clear need to be able to do this effectively.
To date, we are unaware of any systematic review exploring the methods used to assess the usability of electronic information visualisations of patient data for clinical use. This review will summarise the similarities and differences between methods used to assess the usability of electronic information visualisations of patient data for clinical use. We will present the differences between the methods found. Our results will inform best practice when developing new user interfaces to display electronic patient data for clinical use.
The systemic review will be limited by the quality of the data available.
Cumulative Index to Nursing and Allied Health Literature
Excerpta Medica database
Electronic patient record
Medical Literature Analysis and Retrieval System Online
Medical Subject Headings
Department of Health, 2015. Health secretary outlines vision for use of technology across NHS. https://www.gov.uk/government/news/health-secretary-outlines-vision-for-use-of-technology-across-nhs. Accessed 31 March 2016.
West, V.L., Borland, D., Hammond, W.E., (2015). Innovative information visualization of electronic health record data: a systematic review. J Am Med Inform Assoc. http://dx.doi.org/10.1136/amiajnl-2014-002955. Accessed 9 February 2016.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097. doi:10.1371/journal.pmed1000097. Accessed 19 January 2016
Starren J, Johnson SB. An object-orientated taxonomy of medical data presentations. J Am Med Inform Assoc. 2000;7(1):1–20. https://jamia.oxfordjournals.org/. Accessed 4 April 2016
Kopanitsa G, Tsvetkova Z, Veseli H. Analysis of metrics for the usability evaluation of electronic health record systems. Eur Fed Med Inform. 2012; doi:10.3233/978-1-61499-052-9-129. Accessed 8 Feb 2016
Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52:377–84.
This systematic review protocol is funded by the Department of Health and Wellcome Trust through the Health Innovation Challenge Fund.
This publication presents independent research commissioned by the Health Innovation Challenge Fund (HICF-R9-524; WT-103703/Z/14/Z), a parallel funding partnership between the Department of Health and Wellcome Trust. The views expressed in this publication are those of the author(s) and not necessarily those of the Department of Health or Wellcome Trust.
Availability of data and materials
The datasets supporting the conclusions of the systematic review article will be included within the article and in the additional files.
Consent for publication
Consent for publication is not applicable as no individual person’s data is contained in the paper.
AB has a Masters in Ergonomics (Human Factors). LM has a PhD in Ergonomics (Human Factors).
Ethics approval and consent to participate
Name of ethics committee: Oxfordshire Research Ethics Committee C (REC); Confidentiality Advisory Group (CAG)
Reference number: REC: 16/SC/0264; CAG: 16/CAG/0066.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
PRISMA-P checklist.doc. This file contains the completed PRISMA-P checklist. (DOC 102 kb)
Search strategy table.doc. This file contains an example MEDLINE search strategy. (DOCX 21 kb)
PRISMA 2009 flow diagram.doc. This file contains the PRISMA 2009 flow diagram. (DOC 47 kb)
About this article
Cite this article
Burt, A., Morgan, L., Petrinic, T. et al. Usability evaluation methods employed to assess information visualisations of electronically stored patient data for clinical use: a protocol for a systematic review. Syst Rev 6, 148 (2017). https://doi.org/10.1186/s13643-017-0544-1