Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support

Department of Medicine, Division of General Internal Medicine, Mount Sinai School of Medicine, New York, NY, USA.
International Journal of Medical Informatics (Impact Factor: 2). 03/2012; 81(11):761-72. DOI: 10.1016/j.ijmedinf.2012.02.009
Source: PubMed


Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. "Near-live" clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow.
This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving "think-aloud" protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used "near-live" clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods.
In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS.
This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of "think-aloud" protocol analysis and "near-live" clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.

Download full-text


Available from: Dillon Chrimes, Jul 15, 2014
  • Source
    • "However, a usability study with end users is missing to objectively evaluate the tool's real ease of use. Usability testing is essential for determining whether a solution works with target users [25] [26] [27] [28]. Moreover, the authors of PANDA do not detail their design methodology. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Background Users may have difficulty calculating DTI group statistics since they need to master several complex tools that require high user intervention. A tool called DTIStatistics for the automatic and easy calculation of DTI group statistics was developed to reduce analysis times and possible errors. Methods The proposed software was designed by using a user-centred methodology in which we performed an iterative usability evaluation with an expert committee. Once the experts’ requirements were fulfilled, we performed a validation of the final version of DTIStatistics with target users, comparing the execution time of this tool and the standard pipeline normally used. Results Target users needed significantly less time to complete the tasks with DTIStatistics, reducing the analysis time from 1383.78 to 57.2 seconds. They were able to complete all the tasks and barely made errors. Moreover, target users were not able to display the analysis results with the standard pipeline, but when using our tool they only needed 34 seconds. Target users found DTIStatistics easy to learn, use and interact with, and they concluded that they could effectively complete the tasks with it. Additionally, we present example results in the study of depression to demonstrate the validity of DTIStatistics for clinical research. Conclusions DTIStatistics facilitates and significantly automates the calculation of DTI group statistics by reducing the analysis times, which implies lower costs. DTIStatistics is highly applicable in clinical research, as demonstrated by the fact that it is currently being used at the University Hospital, University of Navarra (Spain).
    Full-text · Article · Oct 2014 · Computers in Biology and Medicine
  • Source
    • "In the field of health information technology (HIT), usability research has been identified as an important cognitive challenge for the adoption and meaningful use of HIT by the Office of National Coordinator for Health Information Technology (ONC), which is part of the U.S. Department of Health and Human Services (DHHS), and has become an active area for research, design, and practice in HIT [1] [2] [3] [4] [5] [6] [7] [8]. Methods of usability evaluation have been demonstrated to improve the design and utilization of clinical information systems [9] [10] [11]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Objectives. To systematically review and analyze the current status and characteristics of usability studies in China in the field of information technology in general and in the field of healthcare in particular. Methods. We performed a quantitative literature analysis in three major Chinese academic databases and one English language database using Chinese search terms equivalent to the concept of usability. Results. Six hundred forty-seven publications were selected for analysis. We found that in China the literature on usability in the field of information technology began in 1994 and increased thereafter. The usability definitions from ISO 9241-11:1998 and Nielsen (1993) have been widely recognized and cited. Authors who have published several publications are rare. Fourteen journals have a publishing rate over 1%. Only nine publications about HIT were identified. Discussions. China's usability research started relatively late. There is a lack of organized research teams and dedicated usability journals. High-impact theoretical studies are scarce. On the application side, no original and systematic research frameworks have been developed. The understanding and definition of usability is not well synchronized with international norms. Besides, usability research in HIT is rare. Conclusions. More human and material resources need to be invested in China's usability research, particularly in HIT.
    Full-text · Article · Jun 2014 · BioMed Research International
  • Source
    • "Although our results suggest that incorporating PGx resources into smart PGx-CDS alerts is key to increasing PGx adoption into daily practice and improving patient safety [30] , we learned that presenting PGx information to clinicians is challenging, and that presenting too much information overwhelms participants. Clinically relevant PGx knowledge must be represented in language the physicians find intuitive and easily interpretable [31,32]. In order to avoid alert fatigue, rules for alerts must also be carefully developed and triggered such that recommended actions are clear and facilitate a rapid response [31,33]. "
    [Show abstract] [Hide abstract]
    ABSTRACT: Pharmacogenomics (PGx) is positioned to have a widespread impact on the practice of medicine, yet physician acceptance is low. The presentation of context-specific PGx information, in the form of clinical decision support (CDS) alerts embedded in a computerized provider order entry (CPOE) system, can aid uptake. Usability evaluations can inform optimal design, which, in turn, can spur adoption.
    Preview · Article · May 2014 · International Journal of Medical Informatics
Show more