Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support

Department of Medicine, Division of General Internal Medicine, Mount Sinai School of Medicine, New York, NY, USA.
International Journal of Medical Informatics (Impact Factor: 2). 03/2012; 81(11):761-72. DOI: 10.1016/j.ijmedinf.2012.02.009
Source: PubMed


Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. "Near-live" clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow.
This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving "think-aloud" protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used "near-live" clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods.
In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS.
This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of "think-aloud" protocol analysis and "near-live" clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.

Download full-text


Available from: Dillon Chrimes, Jul 15, 2014
1 Follower
151 Reads
  • Source
    • "However, a usability study with end users is missing to objectively evaluate the tool's real ease of use. Usability testing is essential for determining whether a solution works with target users [25] [26] [27] [28]. Moreover, the authors of PANDA do not detail their design methodology. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Users may have difficulty calculating DTI group statistics since they need to master several complex tools that require high user intervention. A tool called DTIStatistics for the automatic and easy calculation of DTI group statistics was developed to reduce analysis times and possible errors. Methods The proposed software was designed by using a user-centred methodology in which we performed an iterative usability evaluation with an expert committee. Once the experts’ requirements were fulfilled, we performed a validation of the final version of DTIStatistics with target users, comparing the execution time of this tool and the standard pipeline normally used. Results Target users needed significantly less time to complete the tasks with DTIStatistics, reducing the analysis time from 1383.78 to 57.2 seconds. They were able to complete all the tasks and barely made errors. Moreover, target users were not able to display the analysis results with the standard pipeline, but when using our tool they only needed 34 seconds. Target users found DTIStatistics easy to learn, use and interact with, and they concluded that they could effectively complete the tasks with it. Additionally, we present example results in the study of depression to demonstrate the validity of DTIStatistics for clinical research. Conclusions DTIStatistics facilitates and significantly automates the calculation of DTI group statistics by reducing the analysis times, which implies lower costs. DTIStatistics is highly applicable in clinical research, as demonstrated by the fact that it is currently being used at the University Hospital, University of Navarra (Spain).
    Computers in Biology and Medicine 10/2014; 53. DOI:10.1016/j.compbiomed.2014.07.020 · 1.24 Impact Factor
  • Source
    • "In the field of health information technology (HIT), usability research has been identified as an important cognitive challenge for the adoption and meaningful use of HIT by the Office of National Coordinator for Health Information Technology (ONC), which is part of the U.S. Department of Health and Human Services (DHHS), and has become an active area for research, design, and practice in HIT [1] [2] [3] [4] [5] [6] [7] [8]. Methods of usability evaluation have been demonstrated to improve the design and utilization of clinical information systems [9] [10] [11]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Objectives. To systematically review and analyze the current status and characteristics of usability studies in China in the field of information technology in general and in the field of healthcare in particular. Methods. We performed a quantitative literature analysis in three major Chinese academic databases and one English language database using Chinese search terms equivalent to the concept of usability. Results. Six hundred forty-seven publications were selected for analysis. We found that in China the literature on usability in the field of information technology began in 1994 and increased thereafter. The usability definitions from ISO 9241-11:1998 and Nielsen (1993) have been widely recognized and cited. Authors who have published several publications are rare. Fourteen journals have a publishing rate over 1%. Only nine publications about HIT were identified. Discussions. China's usability research started relatively late. There is a lack of organized research teams and dedicated usability journals. High-impact theoretical studies are scarce. On the application side, no original and systematic research frameworks have been developed. The understanding and definition of usability is not well synchronized with international norms. Besides, usability research in HIT is rare. Conclusions. More human and material resources need to be invested in China's usability research, particularly in HIT.
    BioMed Research International 06/2014; 2014:568303. DOI:10.1155/2014/568303 · 2.71 Impact Factor
  • Source
    • "Over time these methods have evolved to better support different contexts of healthcare delivery. A recent variation on usability testing has looked at different levels of usability to include issues beyond the HIT interface itself [18]. Another example of progress on usability testing has been the identification of the usability lifecycle that provides an approach for bridging requirements engineering with evaluation to support more integrated HIT design and evaluation [28]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Context is a key consideration when designing and evaluating health information technology (HIT) and cannot be overstated. Unintended consequences are common post HIT implementation and even well designed technology may not achieve desired outcomes because of contextual issues. While context should be considered in the design and evaluation of health information systems (HISs) there is a shortcoming of empirical research on contextual aspects of HIT. This conference integrates the sociotechnical and Human-Centered-Design (HCD) approaches and showcases current research on context sensitive health informatics. The papers and presentations outlines theories and models for studying contextual issues and insights on how we can better design HIT to accommodate different healthcare contexts.
    Studies in health technology and informatics 08/2013; 194:1-7.
Show more