Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support.

Department of Medicine, Division of General Internal Medicine, Mount Sinai School of Medicine, New York, NY, USA.
International Journal of Medical Informatics (Impact Factor: 2.72). 03/2012; 81(11):761-72. DOI: 10.1016/j.ijmedinf.2012.02.009
Source: PubMed

ABSTRACT Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. "Near-live" clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow.
This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving "think-aloud" protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used "near-live" clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods.
In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS.
This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of "think-aloud" protocol analysis and "near-live" clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.

  • [Show abstract] [Hide abstract]
    ABSTRACT: To understand clinician adoption of CDS tools as this may provide important insights for the implementation and dissemination of future CDS tools.
    Applied Clinical Informatics 01/2014; 5(3):824-35. · 0.39 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Background Users may have difficulty calculating DTI group statistics since they need to master several complex tools that require high user intervention. A tool called DTIStatistics for the automatic and easy calculation of DTI group statistics was developed to reduce analysis times and possible errors. Methods The proposed software was designed by using a user-centred methodology in which we performed an iterative usability evaluation with an expert committee. Once the experts’ requirements were fulfilled, we performed a validation of the final version of DTIStatistics with target users, comparing the execution time of this tool and the standard pipeline normally used. Results Target users needed significantly less time to complete the tasks with DTIStatistics, reducing the analysis time from 1383.78 to 57.2 seconds. They were able to complete all the tasks and barely made errors. Moreover, target users were not able to display the analysis results with the standard pipeline, but when using our tool they only needed 34 seconds. Target users found DTIStatistics easy to learn, use and interact with, and they concluded that they could effectively complete the tasks with it. Additionally, we present example results in the study of depression to demonstrate the validity of DTIStatistics for clinical research. Conclusions DTIStatistics facilitates and significantly automates the calculation of DTI group statistics by reducing the analysis times, which implies lower costs. DTIStatistics is highly applicable in clinical research, as demonstrated by the fact that it is currently being used at the University Hospital, University of Navarra (Spain).
    Computers in Biology and Medicine 10/2014; · 1.48 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Evidence-based solutions for changing health behaviors exist but problems with feasibility, sustainability, and dissemination limit their impact on population-based behavior change and maintenance.
    JMIR mHealth and uHealth. 01/2014; 2(4):e41.


Available from
Jul 15, 2014