Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support

Department of Medicine, Division of General Internal Medicine, Mount Sinai School of Medicine, New York, NY, USA.
International Journal of Medical Informatics (Impact Factor: 2.72). 03/2012; 81(11):761-72. DOI: 10.1016/j.ijmedinf.2012.02.009
Source: PubMed

ABSTRACT Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. "Near-live" clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow.
This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving "think-aloud" protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used "near-live" clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods.
In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS.
This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of "think-aloud" protocol analysis and "near-live" clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.

1 Follower
  • [Show abstract] [Hide abstract]
    ABSTRACT: Objective: To understand clinician adoption of CDS tools as this may provide important insights for the implementation and dissemination of future CDS tools. Materials and Methods: Clinicians (n=168) at a large academic center were randomized into intervention and control arms to assess the impact of strep and pneumonia CDS tools. Intervention arm data were analyzed to examine provider adoption and clinical workflow. Electronic health record data were collected on trigger location, the use of each component and whether an antibiotic, other medication or test was ordered. Frequencies were tabulated and regression analyses were used to determine the association of tool component use and physician orders. Results: The CDS tool was triggered 586 times over the study period. Diagnosis was the most frequent workflow trigger of the CDS tool (57%) as compared to chief complaint (30%) and diagnosis/antibiotic combinations (13%). Conversely, chief complaint was associated with the highest rate (83%) of triggers leading to an initiation of the CDS tool (opening the risk prediction calculator). Similar patterns were noted for initiation of the CDS bundled ordered set and completion of the entire CDS tool pathway. Completion of risk prediction and bundled order set components were associated with lower rates of antibiotic prescribing (OR 0.5; CI 0.2-1.2 and OR 0.5; CI 0.3-0.9, respectively). Discussion: Different CDS trigger points in the clinician user workflow lead to substantial variation in downstream use of the CDS tool components. These variations were important as they were associated with significant differences in antibiotic ordering. Conclusions: These results highlight the importance of workflow integration and flexibility for CDS success.
    Applied Clinical Informatics 01/2014; 5(3):824-35. DOI:10.4338/ACI-2014-04-RA-0043 · 0.39 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Evidence-based solutions for changing health behaviors exist but problems with feasibility, sustainability, and dissemination limit their impact on population-based behavior change and maintenance.
    01/2014; 2(4):e41. DOI:10.2196/mhealth.3307
  • [Show abstract] [Hide abstract]
    ABSTRACT: To examine whether simulation training enhanced self-efficacy of physicians and nurses who work in the outpatient setting to use electronic medical records, and whether the training changed their perceptions about the importance of electronic medical records (EMRs) in helping patients and improving patients' safety. Two-hundred and ninety-three physicians and 94 nurses participated in the study. Participants first attended two computer classroom training sessions on how to use EMRs. Subsequently, the participants attended simulation training and practiced application of EMRs while encountering standardized patients. They answered questionnaires on a seven-point Likert-type scale prior to and immediately after simulation training. The questionnaires assessed their perceptions about the importance of EMRs in helping patients and improving patients' safety and their confidence and preparedness level to use EMRs. The overall self-efficacy of physicians and nurses to use EMRs increased after simulation training as compared to before simulation training. The physicians' and nurses' ratings about importance of EMRs to help patients' and improve patients' safety after simulation training were relatively unchanged compared to the ratings before simulation training. Additionally, participants described simulation training as exceptional, because it was an interactive learning opportunity to use EMRs within a simulated clinical setting with a simulated patient. Simulation training in the current study enhanced physicians' and nurses' level of self-confidence and preparedness to use EMRs. To train health care providers how to use EMRs, simulation training should be considered as an interactive and effective method of teaching prior to implementation of EMRs in medical institutions. Copyright © 2015 Elsevier Ireland Ltd. All rights reserved.
    International Journal of Medical Informatics 02/2015; DOI:10.1016/j.ijmedinf.2015.02.003 · 2.72 Impact Factor