Faculty evaluation of simulation-based modules for assessment of intraoperative decision making

Department of Surgery, Northwestern University, 676 North St. Clair, Suite 650, Chicago, IL 60611, USA.
Surgery (Impact Factor: 3.38). 04/2011; 149(4):534-42. DOI: 10.1016/j.surg.2010.10.010
Source: PubMed


Previous studies using simulation-based curricula have focused largely on technical skills. We developed a set of simulation-based modules that focus on intraoperative decision making. The objective of this study was to conduct a faculty evaluation of: (1) the usefulness of 4 newly developed, simulation-based modules; (2) the curricular need to train and assess intraoperative decision making skills of the residents; and (3) potential for resident benefit.
Simulation-based modules were developed using a cognitive task analysis (CTA) framework. The CTA framework involved faculty interviews focusing on 4 operative tasks that span a range of complexity: (1) creation of small and large bowel stoma, (2) laparoscopic ventral hernia repair, (3) pancreaticojejunostomy, and (4) lymph node biopsy during a mediastinoscopy. An experienced psychologist conducted task-specific, one-on-one interviews with fellowship-trained specialists who perform these operations in their practice. Two faculty were interviewed for each procedure. The interviews lasted a minimum of 1 hour and focused on critical decisions, error prevention, error recognition, and error rescue strategies. The coded interview summaries were used as development guides for the simulation-based learning modules. Each module included locally developed physical models for the simulated operative tasks combined with oral and paper-based questions. The physical models were fabricated in such a way that simulated operative tasks could be performed using standard surgical instruments. To assess the newly developed simulation-based modules, 8 volunteer faculty (50% overlap with the interview pool) participated in a simulation-based exercise during a one-on-one session and then completed an 8-item survey cast on a 5-point Likert agreement scale (1 = strongly disagree, 5 = strongly agree). One of the items was worded negatively to ensure internal consistency. An independent observer recorded faculty session times and assessed faculty engagement in the task (1 = not engaged, 5 = extremely engaged).
On average, faculty spent 60 minutes completing each simulation-based exercise. Over 80% of this time was spent performing the operative tasks as they would during a real-life procedure. Mean engagement rating was 4.9 (maximum 5.0, SD = 0.3). Survey results show strong agreement on the importance of training and assessing intraoperative decision making, and that residents would likely benefit from the simulation-based modules.
We developed 4 high-fidelity simulation-based modules to assess intraoperative decision making. Faculty agree strongly on the importance and need for additional modules.

5 Reads
  • [Show abstract] [Hide abstract]
    ABSTRACT: There is a need to train and evaluate a wide variety of nontechnical surgical skills. The goal of this project was to develop and evaluate a decision-based simulation to assess team skills. The decision-based exercise used our previously validated Laparoscopic Ventral Hernia simulator and a newly developed team evaluation survey. Five teams of 3 surgical residents (N = 15) were tasked with repairing a 10 × 10-cm right upper quadrant hernia. During the simulation, independent observers (N = 6) completed a 6-item survey assessing: (1) work quality; (2) communication; and (3) team effectiveness. After the simulation, team members self-rated their performance by using the same survey. Survey reliability revealed a Cronbach's alpha of r = .811. Significant differences were found when we compared team members' (T) and observers' (O) ratings for communication (T = 4.33/5.00 vs O = 3.00/5.00, P < .01) and work quality (T = 4.33/5.00 vs O = 3.33/5.00, P < .05). The team with the greatest survey ratings was the only group to successfully complete the task. The team evaluation survey had good reliability and correlated with task performance on the simulator. Our current and previous work provides strong evidence that nontechnical and team related skills can be assessed without simulating a crisis situation.
    Surgery 04/2012; 152(2):152-7. DOI:10.1016/j.surg.2012.02.018 · 3.38 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Abstract Aim: There is growing pressure from the government and the public to define proficiency standards for surgical skills. Aim of this study was to estimate the reliability of the Program for Laparoscopic Urological Skills (PLUS) assessment and to set a certification standard for second-year urological residents. Methods: Fifty participants were assessed on performance time and performance quality to investigate the reliability of the PLUS assessment. Generalisability coefficient of 0.8, on a scale of 0 to 1.0, was considered to indicate good reliability for assessment purposes. Pass/fail standards were based on laparoscopic experience: Novices, intermediates, and experts (>100 procedures). The pass/fail standards were investigated for the PLUS performances of 33 second-year urological residents. Results: Fifteen novices, twenty-three intermediates and twelve experts were included. An inter-trial reliability of >0.80 was reached with two trials for each task. Inter-rater reliability of the quality measurements was 0.79 for two judges. Pass/fail scores were determined for the novice/intermediate boundary and the intermediate/expert boundary. Pass rates for second-year residents were 63.64% and 9.09%, respectively. Conclusion: The PLUS assessment is reliable for setting a certification standard for second-year urological residents that serves as a starting point for residents to proceed to the next level of laparoscopic competency.
    Minimally invasive therapy & allied technologies: MITAT: official journal of the Society for Minimally Invasive Therapy 05/2012; 22(1). DOI:10.3109/13645706.2012.686918 · 1.27 Impact Factor
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Surgical simulation applications have been largely limited to the acquisition and assessment of technical skills. Current teaching and assessment of surgical judgment is nonsystematic and prone to error. Interest in methods to enhance the acquisition and assessment of knowledge-based (judgment) skills for intraoperative decision making has led to the application of cognitive task analysis (CTA) and human error assessment to facilitate this process. CTA-based delineation of the steps and hazards of a surgical procedure creates a structured process to teach and assess expert surgical judgment and improves trainees' operative planning, hazard recognition, error prevention, and error recovery when coupled with low-fidelity, synthesized simulation models for open and laparoscopic surgery. Web-based simulation applications facilitate curricular learning (rules-guided skills), allow cognitive rehearsal of procedures, and are accessible independent of location and time. Simulation applications that facilitate the assessment and learning of expert intraoperative judgment should include a consensus-derived outline based on CTA of the operative steps and potential points of risk for each surgical procedure; the ability to detect the situational awareness of the performer and the options considered to avoid error at critical steps; an assessment (scoring) of options considered or attempted; immediate evaluation feedback to inform improved performance; and a program of deliberate practice in which progressively more challenging scenarios can be introduced, based on the trainee's demonstrated skills. High-fidelity simulators currently lack these essential components, and future simulation-assisted teaching and assessment of surgical judgment skills are likely to employ low-fidelity simulators coupled to Web-based instruction.
    Academic medicine: journal of the Association of American Medical Colleges 05/2012; 87(7):934-41. DOI:10.1097/ACM.0b013e3182583248 · 2.93 Impact Factor
Show more