Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support

Department of Medicine, Division of General Internal Medicine, Mount Sinai School of Medicine, New York, NY, USA.
International Journal of Medical Informatics (Impact Factor: 2.72). 03/2012; 81(11):761-72. DOI: 10.1016/j.ijmedinf.2012.02.009
Source: PubMed

ABSTRACT Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. "Near-live" clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow.
This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving "think-aloud" protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used "near-live" clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods.
In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS.
This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of "think-aloud" protocol analysis and "near-live" clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.

  • [Show abstract] [Hide abstract]
    ABSTRACT: Objective: To understand clinician adoption of CDS tools as this may provide important insights for the implementation and dissemination of future CDS tools. Materials and Methods: Clinicians (n=168) at a large academic center were randomized into intervention and control arms to assess the impact of strep and pneumonia CDS tools. Intervention arm data were analyzed to examine provider adoption and clinical workflow. Electronic health record data were collected on trigger location, the use of each component and whether an antibiotic, other medication or test was ordered. Frequencies were tabulated and regression analyses were used to determine the association of tool component use and physician orders. Results: The CDS tool was triggered 586 times over the study period. Diagnosis was the most frequent workflow trigger of the CDS tool (57%) as compared to chief complaint (30%) and diagnosis/antibiotic combinations (13%). Conversely, chief complaint was associated with the highest rate (83%) of triggers leading to an initiation of the CDS tool (opening the risk prediction calculator). Similar patterns were noted for initiation of the CDS bundled ordered set and completion of the entire CDS tool pathway. Completion of risk prediction and bundled order set components were associated with lower rates of antibiotic prescribing (OR 0.5; CI 0.2-1.2 and OR 0.5; CI 0.3-0.9, respectively). Discussion: Different CDS trigger points in the clinician user workflow lead to substantial variation in downstream use of the CDS tool components. These variations were important as they were associated with significant differences in antibiotic ordering. Conclusions: These results highlight the importance of workflow integration and flexibility for CDS success.
    Applied Clinical Informatics 01/2014; 5(3):824-35. DOI:10.4338/ACI-2014-04-RA-0043 · 0.39 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Background Users may have difficulty calculating DTI group statistics since they need to master several complex tools that require high user intervention. A tool called DTIStatistics for the automatic and easy calculation of DTI group statistics was developed to reduce analysis times and possible errors. Methods The proposed software was designed by using a user-centred methodology in which we performed an iterative usability evaluation with an expert committee. Once the experts’ requirements were fulfilled, we performed a validation of the final version of DTIStatistics with target users, comparing the execution time of this tool and the standard pipeline normally used. Results Target users needed significantly less time to complete the tasks with DTIStatistics, reducing the analysis time from 1383.78 to 57.2 seconds. They were able to complete all the tasks and barely made errors. Moreover, target users were not able to display the analysis results with the standard pipeline, but when using our tool they only needed 34 seconds. Target users found DTIStatistics easy to learn, use and interact with, and they concluded that they could effectively complete the tasks with it. Additionally, we present example results in the study of depression to demonstrate the validity of DTIStatistics for clinical research. Conclusions DTIStatistics facilitates and significantly automates the calculation of DTI group statistics by reducing the analysis times, which implies lower costs. DTIStatistics is highly applicable in clinical research, as demonstrated by the fact that it is currently being used at the University Hospital, University of Navarra (Spain).
    Computers in Biology and Medicine 10/2014; 53. DOI:10.1016/j.compbiomed.2014.07.020 · 1.48 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Evidence-based solutions for changing health behaviors exist but problems with feasibility, sustainability, and dissemination limit their impact on population-based behavior change and maintenance.
    01/2014; 2(4):e41. DOI:10.2196/mhealth.3307


Available from
Jul 15, 2014