User-centered design and the development of patient decision aids: protocol for a systematic review
- Holly O Witteman1, 2, 3Email author,
- Selma Chipenda Dansokho1,
- Heather Colquhoun4,
- Angela Coulter5,
- Michèle Dugas1,
- Angela Fagerlin6, 7,
- Anik MC Giguere1, 2, 8,
- Sholom Glouberman9,
- Lynne Haslett10,
- Aubri Hoffman11,
- Noah Ivers12, 13,
- France Légaré2, 3,
- Jean Légaré14,
- Carrie Levin15,
- Karli Lopez16,
- Victor M Montori17,
- Thierry Provencher1,
- Jean-Sébastien Renaud1, 2,
- Kerri Sparling18,
- Dawn Stacey19, 20,
- Gratianne Vaisson1,
- Robert J Volk21 and
- William Witteman3
© Witteman et al.; licensee BioMed Central. 2015
Received: 18 October 2014
Accepted: 29 December 2014
Published: 26 January 2015
Providing patient-centered care requires that patients partner in their personal health-care decisions to the full extent desired. Patient decision aids facilitate processes of shared decision-making between patients and their clinicians by presenting relevant scientific information in balanced, understandable ways, helping clarify patients’ goals, and guiding decision-making processes. Although international standards stipulate that patients and clinicians should be involved in decision aid development, little is known about how such involvement currently occurs, let alone best practices. This systematic review consisting of three interlinked subreviews seeks to describe current practices of user involvement in the development of patient decision aids, compare these to practices of user-centered design, and identify promising strategies.
A research team that includes patient and clinician representatives, decision aid developers, and systematic review method experts will guide this review according to the Cochrane Handbook and PRISMA reporting guidelines. A medical librarian will hand search key references and use a peer-reviewed search strategy to search MEDLINE, EMBASE, PubMed, Web of Science, the Cochrane Library, the ACM library, IEEE Xplore, and Google Scholar. We will identify articles across all languages and years describing the development or evaluation of a patient decision aid, or the application of user-centered design or human-centered design to tools intended for patient use. Two independent reviewers will assess article eligibility and extract data into a matrix using a structured pilot-tested form based on a conceptual framework of user-centered design. We will synthesize evidence to describe how research teams have included users in their development process and compare these practices to user-centered design methods. If data permit, we will develop a measure of the user-centeredness of development processes and identify practices that are likely to be optimal.
This systematic review will provide evidence of current practices to inform approaches for involving patients and other stakeholders in the development of patient decision aids. We anticipate that the results will help move towards the establishment of best practices for the development of patient-centered tools and, in turn, help improve the experiences of people who face difficult health decisions.
Systematic review registration
KeywordsPatient decision aids Decision support Shared decision-making Patient education Counseling User-centered design Human-centered design Patient partnership Stakeholder engagement Implementation Knowledge translation Patient-centered care
Patients are increasingly becoming involved in health research, not only as research participants but as partners with valuable expertise, perspectives, and insights for setting agendas, planning and carrying out projects, interpreting findings, and translating new knowledge to patient communities [1, 2]. Patient partnership in research teams is increasingly encouraged or required by funding organizations [3–6]. However, there are few empirically based best practices for research partnerships between patients, other stakeholders, and researchers.
The question of how to best involve patients in research is especially relevant in the development of patient decision aids. Patient decision aids are structured tools, often booklets or websites, that aim to provide unbiased, evidence-based information and guidance to patients making health decisions . Unlike more general health education materials such as information leaflets, decision aids specifically support decision-making by making the decision explicit, providing balanced information on benefits and harms of options, and helping patients clarify what is most important in their own circumstances. They are intended to be used by patients to complement information and counseling from a health-care professional in the process of shared decision-making  and provide a means for clinicians and patients to collaboratively incorporate their expertise, insights, and views in order to make evidence-based health decisions that are aligned with patients’ preferences [9, 10].
The International Patient Decision Aid Standards (IPDAS) Collaboration stipulates that the development of a patient decision aid should follow a systematic process and should involve consultation with patients and clinicians. However, due to the lack of a robust evidence base from which to draw conclusions about best practices, practical guidance is minimal and vague [11–14]. Notably, only about half of patient decision aids included in the most recent Cochrane review of treatment and screening patient decision aids  reported having involved patients in their development process in some way .
Leaders in the field have offered insights based on their experiences of developing decision aids, including the importance of consulting with patients and other stakeholders [15–17], but there is little empirical evidence available about methods for putting patients at the center of the process. Accordingly, a recent update of the evidence base by the International Patient Decision Aid Standards Collaboration called for greater research into methods for patient involvement in decision aid development. Specifically, the relevant chapter in the updated standards states: “More guidance is needed to inform patient decision aid alpha- and beta-tests, including user-centered design methods […]. The process of designing the patient decision aid remains rather subjective” .
Our overall aim in this project is to ultimately improve the effectiveness, usability, and uptake of patient decision aids by identifying effective methods for involving patients and other stakeholders in their development. To accomplish this, we have identified three specific aims for this systematic review: (1) To describe how patients and other stakeholders, including clinicians, have and have not been involved in the development of patient decision aids. (2) To compare methods used for engaging patients and other stakeholders in decision aid development with methods of user-centered design. (3) To identify promising strategies for involving patients and other stakeholders in the development of patient decision aids.
This framework rests on the idea that a system, service, or product is most likely to fulfill user needs when its development process is based on iterative cycles in which potential users are consulted early and often. In the case of patient decision aids, lack of a fully iterative feedback loop may result in decision aids that are not optimized to meet people’s needs. When users are not able to critique a design until it is far along in the development process, it may be too late to make certain types of changes given time and cost constraints. Research in the field of user-centered design in non-health contexts offers some guidance regarding, for example, how many potential users might need to participate in a software development process to ensure that a product is fully usable [34–36]. However, it is unknown whether or not these findings are applicable to patient-oriented tools such as decision aids and whether or not the framework might serve as a useful guide to optimize the development of such tools.
This conceptual framework will serve two purposes in this review. Its primary purpose will be to structure our data extraction process. We will ensure that our data extraction form captures each element in the framework. Its secondary purpose will be to help generate and structure hypotheses for exploratory analyses.
Inclusion and exclusion criteria
We will include three potentially overlapping groups of articles (see Figure 2) in order to conduct our planned syntheses and comparisons.
The first group of articles will be those that describe the development of a decision aid intended to support a patient’s health decision or the decision of a surrogate regarding a patient. We will use data from these articles in addressing all three research questions. To be eligible for inclusion in this group, articles must explicitly describe at least one step in the development process of a patient decision aid. We will exclude articles that do not meet this criterion.
The second group of articles will be those that explicitly describe the application of user/human-centered design in the development of patient-oriented tools. We will use data from these articles to address research questions 2 and 3. These articles may describe patient-oriented tools that go beyond decision aids, e.g., tools for education, social support, or self-management. This includes health management tools used by members of the public who are not necessarily active patients in the health-care system. We will exclude related articles about applying user/human-centered design to tools intended for use solely by clinicians.
The third group of articles will be those that describe the evaluation of a patient decision aid. We will use data from these articles to address research question 3. Articles about evaluation of decision aids may describe preliminary testing phases, feasibility and acceptability studies, pilot studies, randomized controlled trials, or other types of evaluations of a developed patient decision aid. This project is expanding upon an existing systematic review, namely, the Cochrane review of decision aids for screening and treatment decisions , which is led by a member of our research team (DS). This review will necessarily include all articles within the Cochrane review, as well as others. These other articles will largely describe quantitative studies, but we will also include some qualitative studies or studies that include both qualitative and quantitative results, for example, those that describe the evaluation of the usability of a prototype patient decision aid. Evaluation studies may focus on the efficacy of a decision aid (e.g., capacity to increase patient knowledge or decrease decisional conflict within the context of a formal study), its actual or potential implementation (e.g., was it successfully integrated into routine care, does a feasibility study suggest it is implementable), and/or its effectiveness (e.g., does it show desired effects in real-world conditions). We will exclude articles that do not meet this criterion.
Articles will be screened for inclusion by two independent reviewers who will first examine titles and abstracts to assess potential relevance. All articles deemed potentially relevant based on their title and abstract will then be reviewed in full by two independent reviewers. Any discrepancies in inclusion and exclusion at any stage of this process will be addressed through regular discussions among the principal investigator, research staff, and medical librarian (HOW, SCD, MD, TP, GV, WW).
We will track articles’ inclusion and exclusion for reporting and will link together articles published separately about the same decision aid. This linking will account for the fact that it is common practice for research teams to publish a preliminary article describing the development and perhaps initial testing of a decision aid, followed by a second article one or more years later describing the evaluation of that decision aid.
We will perform a systematic literature search in MEDLINE, EMBASE, PubMed, Web of Science, the Cochrane Library, the ACM library, and IEEE Xplore. We also use Google Scholar for citation analysis of certain key papers, such as the 2006 International Patient Decision Aid Standards. An overview of our search strategy is shown in Figure 2.
We anticipate considerable overlap between our first and third groups, that is, articles describing the development of a patient decision aid and articles describing the evaluation of patient decision aids. For example, it is common for decision aid developers to publish an article describing the development and feasibility testing of a decision aid. Therefore, to identify articles in the first and third groups, we will seek articles in the union of these groups. We will search using the term “development” both as a free-text term and as a controlled-vocabulary term (MeSH, EMTREE), in conjunction with terms distinguishing the references we wanted from those describing biological development. We will also use other search terms such as usability and sets of terms such as test, study, or evaluation in close conjunction with terms like pilot or feasibility. To identify articles describing evaluation studies that were not randomized controlled trials, we will use a peer-reviewed, published search string that filters for observational studies . We will search for articles in the intersection of any of these and an adapted version of the search string used to identify decision aids in the Cochrane review of decision aids . See a sample search string in Additional file 1 that shows the entire strategy described above. We will also examine each article included in the Cochrane review of decision aids for screening or treatment decisions  and search its references to identify previous articles published by the authors that might have described the development process of the decision aid.
To identify articles in the second group (application of user-centered design to tools intended for patient use), we will search for the explicit use of the free-text terms “user-centred design”, “user-centered design”, “human-centred design”, or “human-centered design”, because for this group we will be seeking articles in which the authors clearly identify their development method as user/human-centered design.
Prior to conducting searches, our search strategy will be peer-reviewed by another medical librarian not on our study team.
Data abstraction and validation
Data from each article will be abstracted by two independent, trained research team members using a standardized and pilot-tested data extraction form. The preliminary form will be based on our conceptual framework of user-centered design and relevant standards. Once we have established a preliminary form, we will conduct semi-structured consultations with approximately 12 key informants whom we believe can comment on our data extraction plan and potentially make suggestions for its improvement. These key informants may be researchers who have expertise in the area of development of patient-centered tools such as decision aids, or they may be patients or stakeholders whom we believe have a perspective not already represented on this project team. We will aim to consult with people who will bring a diverse set of perspectives and expertise. We will also pilot test the form with randomly selected articles from each of the three groups. We will iteratively revise the form, taking into account key informants’ comments and the results of pilot tests.
Once the form is finalized, pairs of data abstractors will independently extract data from all articles. Lack of agreements will be resolved through discussion until consensus is reached, either with the principal investigator and research staff (HOW, SCD, MD, TP, GV, WW), with the project’s steering committee (HOW, AF, AG, SG, FL, KS, RJV), or with the full team (all authors).
From all articles, we will extract key factors grounded in our conceptual framework of user-centered design, including: (i) whether patients and other stakeholders were involved in the development and, if yes, (ii) how they were defined (e.g., people who had previously faced the decision, people who might potentially face this decision, patients who were actually facing this decision, caregivers, clinicians who were not part of the research team), (iii) how were they identified and recruited, (iv) what information was collected about their needs and personal contexts, (v) how many were recruited, (vi) how often were patients and stakeholders consulted in the development process (i.e., were there iterative cycles of consultation, and, if yes, how many cycles), (vii) at what point(s) in the process were they involved (i.e., at the beginning during idea generation, once a fully developed prototype was prepared), (viii) what did their involvement consist of (i.e., were they observed interacting with the decision aid in a naturalistic fashion or were they asked to speculate on how they might use it), (ix) types of feedback sought (comprehensiveness or appropriateness of content, format, other), (x) how was their feedback incorporated into the design, and (xi) were members of potentially underserved populations involved (for example, patients with limited access to health care, people with lower health literacy, patients who have other health conditions that might complicate the decision under study, etc.). We will also record secondary data about whether or not each decision aid cited a relevant guideline, and the theory(ies) or framework(s) underpinning each decision aid. In addition, from articles describing the evaluation of a decision aid, we will extract information about the type of evaluation, specifically, whether or not outcomes such as feasibility, acceptability, and usability were reported, and if so, what metrics were used and what was reported according to those metrics. Any such additional evaluation data will be added to existing evaluation data from the Cochrane review of patient decision aids .
Throughout the data extraction process, research assistants will enter all data extracted into a data matrix structured according to our conceptual framework of user-centered design. In the matrix, we will match up articles describing the development of a patient decision aid with any evaluation data for that decision aid. We will track the source of each row in the matrix, i.e., whether it is a patient decision aid or a patient-oriented tool (for example, a self-management tool that is not oriented around decisions) developed with a user-centered design approach.
Because of the heterogeneity of reporting about development processes—namely, some research teams do not describe their processes at all, and, of those that do, the detail given varies considerably—and because we wish to ensure that we have understood each team’s work correctly, we will contact authors to review the data we have extracted about their study and collect any data that they did not or were unable to report in their publication. We will track which data were abstracted from the article and which were provided directly by authors.
The quality of each article will be appraised by two independent reviewers using adapted established quality criteria for scoring and appraising studies in a mixed method review . We will track and record appraisal for reporting purposes.
Evidence synthesis and analysis
To address our first research question (when developing patient decision aids, how are patients and other stakeholders currently involved?), we will assess and report frequency of use of different practices used in patient decision aid development, including the extent to which methods used in decision aid development align with methods of user-centered design by examining descriptive frequencies of practices within our conceptual framework. For example, we will determine the proportion of patient decision aids developed with involvement by patients who have previously faced the decision. This will allow us to develop an overall picture of practices used (or not used). If sufficient data exist for this purpose, we will also explore descriptively how practices may have changed over time.
To address our second research question (how do development practices for patient decision aids compare to those within user-centered design?), we will also assess and report the same descriptive frequencies for the development of other tools designed for patient use where authors of published reports explicitly described the development approach as user-centered design or human-centered design. We will explore potential differences by comparing and contrasting the development processes of patient decision aids with the development of other patient-centered tools explicitly employing user-centered design. For these comparisons, any patient decision aids that explicitly report using user-centered design will be treated separately, depending on the question addressed.
To address our third research question (which development practices are associated with better outcomes?), we will use a two-pronged analytic approach. First, in addition to examining descriptive frequencies for our first research question, if the data permit such development, we will also seek to develop a measure of patient- and stakeholder-centeredness of patient decision aid development processes. Second, if the data permit, we will also explore whether some strategies for patient involvement are associated with better outcomes, for example, increased knowledge scores, indications of better usability, or improved clarity about one’s values relevant to a decision. If it is possible to develop a measure as described above, we will use that measure as the independent variable in these latter analyses. Otherwise, we will conduct exploratory analyses using multiple independent variables selected primarily on the basis of their frequency of use. The primary outcomes for these analyses will be the same as those in the Cochrane review of patient decision aids, including patient knowledge and clarity about one’s values . A secondary outcome of particular relevance to this review will be indications of usability. Given that this is a relatively new outcome in health research, the method of synthesis for this outcome is difficult to fully specify in advance, as it will depend on what is reported in the included studies. We anticipate synthesizing whether or not usability was reported at all, and if yes, metrics of ease of use (for example, the percentage of users rating a tool as easy to use), whether or not lists of usability problems were generated and addressed, and any reported validated measures of usability. This synthesis plan may be adapted according to reported outcomes in the identified articles.
With this systematic review, we aim to provide empirical evidence to help guide the inclusion of patients and other stakeholders in the development processes of patient decision aids with the ultimate goal of improving their effectiveness, usability, and uptake. The framework of user-centered design offers a robust and yet underused framework for examining current and potential methods for developing tools for patient use. By working to improve methods for involving patient partners and other stakeholders in the development process, we seek to help ensure that research responds to patient priorities and concerns and makes the best possible use of patients’ and other stakeholders’ time and expertise, as well as limited research funds. If data allow, this project may also contribute a measure of the user-centeredness of decision aid development that could be used to evaluate the quality of future patient decision aids’ development processes.
It is important to note that while this project will be conducted in the context of decision aid development, we anticipate that our findings may also apply more broadly to research involving development of other patient- or caregiver-oriented tools, services, and systems. Such other tools may be designed for purposes other than decision support, for example, patient education materials and self-management tools such as chronic disease management systems, applications for communication with health-care teams, and patient portals for electronic medical records. Development processes for these could also benefit significantly from optimal methods for patient and other stakeholder involvement. Because user-centered design is rooted in the field of human-computer interaction, we anticipate that our findings may be especially pertinent to those developing technology-oriented applications such as patient portals and mobile or online health applications. Providing this kind of guidance to improve the development processes of such tools, services, and systems may help increase their patient-centeredness.
Association for computing machinery
Institute of Electrical and Electronics Engineers
International patient decision aid standards
Patient-Centered Outcomes Research Institute
Preferred reporting items for systematic reviews and meta-analyses.
This study was funded by the Patient-Centered Outcomes Research Institute (PCORI): ME-1306-03174. PCORI had no role in determining the study design, the plans for data collection or analysis, the decision to publish, nor the preparation of this manuscript. HOW is funded by a Research Scholar Junior 1 Career Development Award by the Fonds de Recherche du Québec—Santé. NI is funded by a New Investigator Award by the Canadian Institutes of Health Research as well as a New Investigator Award NIA from the Department of Family and Community Medicine, University of Toronto. FL is funded by a Tier 2 Canada Research Chair in Implementation of Shared Decision Making. DS is funded by a University of Ottawa Research Chair in Knowledge Translation to Patients. The authors gratefully acknowledge the contributions of Issa Bado to this project during a research internship.
- Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, et al.: Systematic review of involving patients in the planning and development of health care.BMJ. 2002, 325:1263. 10.1136/bmj.325.7375.1263View ArticlePubMedPubMed CentralGoogle Scholar
- Longtin Y, Sax H, Leape LL, Sheridan SE, Donaldson L, Pittet D: Patient participation: current knowledge and applicability to patient safety.Mayo Clin Proc. 2010, 85:53–62. 10.4065/mcp.2009.0248View ArticlePubMedPubMed CentralGoogle Scholar
- Topic of conversation: Den Haag. Netherlands: Dutch organization for health and innovation in healthcare; 2010:1–17.Google Scholar
- Collier R: Federal government unveils patient-oriented research strategy.Can Med Assoc J. 2011, 183:E993–4. 10.1503/cmaj.109-3978View ArticleGoogle Scholar
- Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al.: Mapping the impact of patient and public involvement on health and social care research: a systematic review.Health Expect. 2014, 17:637–50. 10.1111/j.1369-7625.2012.00795.xView ArticlePubMedGoogle Scholar
- Frank L, Basch E, Selby JV: The PCORI perspective on patient-centered outcomes research.JAMA. 2014, 312:1513–14. 10.1001/jama.2014.11100View ArticlePubMedGoogle Scholar
- Stacey D, Legare F, Col NF, Bennett CL, Barry MJ, Eden KB, et al.: Decision aids for people facing health treatment or screening decisions.Cochrane Database Syst Rev. 2014., 1: CD001431Google Scholar
- Makoul G, Clayman ML: An integrative model of shared decision making in medical encounters.Patient Educ Couns. 2006, 60:301–12. 10.1016/j.pec.2005.06.010View ArticlePubMedGoogle Scholar
- Barry MJ, Edgman-Levitan S: Shared decision making—pinnacle of patient-centered care.N Engl J Med. 2012, 366:780–1. 10.1056/NEJMp1109283View ArticlePubMedGoogle Scholar
- Mulley AG, Trimble C, Elwyn G: Stop the silent misdiagnosis: patients’ preferences matter.BMJ. 2012, 345:e6572–2. 10.1136/bmj.e6572View ArticlePubMedGoogle Scholar
- Elwyn G, O’Connor A, Stacey D, Volk RJ, Edwards A, Coulter A, et al.: Developing a quality criteria framework for patient decision aids: online international Delphi consensus process.BMJ. 2006, 333:417. 10.1136/bmj.38926.629329.AEView ArticlePubMedPubMed CentralGoogle Scholar
- Elwyn G, O’Connor AM, Bennett C, Newcombe RG, Politi MC, Durand M-A, et al.: Assessing the quality of decision support technologies using the International Patient Decision Aid Standards instrument (IPDASi).PLoS One. 2009, 4:e4705. 10.1371/journal.pone.0004705View ArticlePubMedPubMed CentralGoogle Scholar
- Coulter A, Kryworuchko J, Mullen PD, Ng CJ, Stillwell D, van der Weijden T: Chapter A: using a systematic development process. In 2012 update of the international patient decision Aid standards (IPDAS) Collaboration’s background document Edited by: Volk RJ, Llewellyn-Thomas HA. 2012, 1–16. Available at: http://ipdas.ohri.ca/IPDAS-Chapter-A.pdfGoogle Scholar
- Coulter A, Stilwell D, Kryworuchko J, Mullen PD, Ng CJ, van der Weijden T: A systematic development process for patient decision aids.BMC Med Inform Decis Mak. 2013, 13:S2.View ArticlePubMedPubMed CentralGoogle Scholar
- Elwyn G, Kreuwel I, Durand M-A, Sivell S, Joseph-Williams N, Evans R, et al.: How to develop web-based decision support interventions for patients: a process map.Patient Educ Couns. 2011, 82:260–5. 10.1016/j.pec.2010.04.034View ArticlePubMedGoogle Scholar
- O’Connor AM, Tugwell P, Wells GA, Elmslie T, Jolly E, Hollingworth G, et al.: A decision aid for women considering hormone therapy after menopause: decision support framework and evaluation.Patient Educ Couns. 1998, 33:267–79. 10.1016/S0738-3991(98)00026-3View ArticlePubMedGoogle Scholar
- Montori VM, Breslin MA, Maleska M, Weymiller AJ: Creating a conversation: insights from the development of a decision aid.PLoS Med. 2007, 4:e233. 10.1371/journal.pmed.0040233View ArticlePubMedPubMed CentralGoogle Scholar
- Abras C, Maloney-Krichmar D, Preece J: User-centered design. In Encyclopedia of human-computer interaction. Edited by: Bainbridge W. Thousand Oaks: Sage Publications; 2004:1–14.Google Scholar
- Kelley T, Littman J: The art of innovation: lessons in creativity from IDEO, America’s leading design firm. New York: Doubleday; 2001.Google Scholar
- Norman DA: The design of everyday things. New York: Basic Books; 2002.Google Scholar
- Shneiderman B, Plaisant C: Designing the user interface. 5th edition. Reading, MA: Addison-Wesley Longman; 2009.Google Scholar
- Iso DIS: 9241–210:2010: ergonomics of human-system interaction—part 210: human-centred design for interactive systems (formerly known as 13407). Switzerland: International Standards Organization; 2010.Google Scholar
- Searl MM, Borgi L, Chemali Z: It is time to talk about people: a human-centered healthcare system.Health Research Policy and Systems. 2010, 8:35. 10.1186/1478-4505-8-35View ArticlePubMedPubMed CentralGoogle Scholar
- Wolpin S, Stewart M: A deliberate and rigorous approach to development of patient-centered technologies.Semin Oncol Nurs. 2011, 27:183–91. 10.1016/j.soncn.2011.04.003View ArticlePubMedPubMed CentralGoogle Scholar
- Elkin PL: Human factors engineering in HI: so what? who cares? and what’s in it for you?Healthc Inform Res. 2012, 18:237. 10.4258/hir.2012.18.4.237View ArticlePubMedPubMed CentralGoogle Scholar
- Schaeffer M, Moore BJ: User-centered design: clothing the EMR emperor. In Proceedings of the 2012 symposium on human factors and ergonomics in health care. Human Factors and Ergonomics Society; 2012:166–172.Google Scholar
- Gurses AP, Ozok AA, Pronovost PJ: Time to accelerate integration of human factors and ergonomics in patient safety.BMJ Quality & Safety. 2012, 21:347–51. 10.1136/bmjqs-2011-000421View ArticleGoogle Scholar
- Garrett JJ: Elements of user experience. 2nd edition. New Riders: Berkeley, CA; 2011.Google Scholar
- Tullis T, Albert W: Measuring the user experience. Kindle. Waltham, MA: Morgan Kaufmann; 2010.Google Scholar
- Kuniavsky M, Moed A, Goodman E: Observing the user experience. 2nd edition. Waltham, MA: Morgan Kaufmann; 2012.Google Scholar
- Gould JD, Lewis C: Designing for usability: key principles and what designers think.Commun ACM. 1985, 28:300–11. 10.1145/3166.3170View ArticleGoogle Scholar
- Nielsen J: The usability engineering life cycle.Computer. 1992, 25:12–22.View ArticleGoogle Scholar
- Mao J-Y, Vredenburg K, Smith PW, Carey T: The state of user-centered design practice.Commun ACM. 2005, 48:105–9.View ArticleGoogle Scholar
- Faulkner L: Beyond the five-user assumption: benefits of increased sample sizes in usability testing.Behav Res Methods Instrum Comput. 2003, 35:379–83. 10.3758/BF03195514View ArticlePubMedGoogle Scholar
- Lindgaard G, Chattratichart J: Usability testing: what have we overlooked? In CHI ’07: Proceedings of the SIGCHI conference on Human factors in computing systems. New York, NY: ACM; 2007.Google Scholar
- Bastien JMC: Usability testing: a review of some methodological and technical aspects of the method.Int J Med Inform. 2010, 79:e18–23. 10.1016/j.ijmedinf.2008.12.004View ArticlePubMedGoogle Scholar
- Observational studies methodology filter. http://www.sign.ac.uk/methodology/filters.html#obs
- Pluye P, Gagnon M-P, Griffiths F, Johnson-Lafleur J: A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in mixed studies reviews.Int J Nurs Stud. 2009, 46:529–46. 10.1016/j.ijnurstu.2009.01.009View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.