Rose Hatala's research while affiliated with University of British Columbia - Vancouver and other places

Publications (118)

Article
Full-text available
The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for...
Article
Introduction: Postgraduate competency-based medical education has been implemented with programmatic assessment that relies on entrustment-based ratings. Yet, in less procedurally oriented specialties such as internal medicine, the relationship between entrustment and supervision remains unclear. We undertook the current study to address how inter...
Article
Context: Residents play a pivotal role in medical students' clinical education. From a feedback lens, the near-peer relationship between student and resident holds the potential to foster an educational alliance that could influence learning. We undertook the current qualitative study to explore medical students' perceptions of feedback experience...
Article
Background: The clinical teaching unit is a widespread clinical training model that requires reform to prepare physicians for practice in the 21st century. In this systematic review, we aimed to identify evidence-based practices in internal medicine clinical teaching units that contribute to improved clinical education and health care delivery. M...
Article
The adoption of entrustment ratings in medical education is based on a seemingly simple premise: to align workplace-based supervision with resident assessment. Yet it has been difficult to operationalize this concept. Entrustment rating forms combine numeric scales with comments and are embedded in a programmatic assessment framework, which encoura...
Article
To establish a research and development agenda for Entrustable Professional Activities (EPAs) for the coming decade, the authors, all active in this area of investigation, reviewed recent research papers, seeking recommendations for future research. They pooled their knowledge and experience to identify 3 levels of potential research and developmen...
Article
Full-text available
Introduction Workplace-based assessment in competency-based medical education employs entrustment-supervision scales to suggest trainee competence. However, clinical supervision involves many factors and entrustment decision-making likely reflects more than trainee competence. We do not fully understand how a supervisor’s impression of trainee comp...
Article
Conceptualizations of workplace learning have moved from knowledge acquisition to learning as participation in the practices and cultures of the workplace environment. Along with this has come an appreciation of applicability of sociocultural learning theories, which frame learning as occurring within "communities of practice" or learning being "si...
Article
Full-text available
Competency-based medical education and programmatic assessment intend to increase the opportunities for meaningful feedback, yet these conversations remain elusive. By comparing resident and faculty perceptions of feedback opportunities within one internal medicine residency training program, we sought to understand whether and how principles under...
Article
Statement: Learning curves are used in health professions education to graphically represent paths to competence and expertise. However, research using learning curves often omit important information. The authors conducted a systematic review of the reporting quality of learning curves in simulation-based education research to identify specific a...
Article
Background Ongoing learning in complex clinical environments requires health professionals to assess their own performance, manage their learning, and modify their practices based on self‐monitored progress. Self‐regulated learning (SRL) theory suggests that while learners may be capable of such learning, they often need guidance to enact it effect...
Article
Rationale and aim: In virtual dissection, three-dimensional computed tomography scans are viewed on a near-life size virtual dissection table and through touchscreen technology, students work together to manipulate the data to perform their dissection. The purpose of this study was to develop a Virtual Dissection Curriculum for first year medical...
Chapter
The choice of outcome measure for simulation studies is a crucial element of the research design, for without careful forethought and planning how can we be confident that we are measuring what we intend to measure? In this chapter, we follow on from the concepts introduced in Chap. 25, outlining the key elements in developing and examining the val...
Article
The complex and dynamic nature of the clinical environment often requires health professionals to assess their own performance, manage their learning, and modify their practices based on self-monitored progress. Self-regulated learning studies suggests that while learners may be capable of such in situ learning, they often need guidance to enact it...
Article
Full-text available
Background: Radiology integration into medical anatomy courses is well established, but there is a paucity of literature on integrating virtual dissection into cadaveric dissection laboratories. Virtual dissection is the digital dissection of medical images on touchscreen anatomy visualization tables. The purpose of this pilot study was to investi...
Article
The implementation of Entrustable Professional Activities has led to the simultaneous development of assessment based on a supervisor’s entrustment of a learner to perform these activities without supervision. While entrustment may be intuitive when we consider the direct observation of a procedural task, the current implementation of rating scales...
Article
Full-text available
Learning curves can support a competency-based approach to assessment for learning. When interpreting repeated assessment data displayed as learning curves, a key assessment question is: “How well is each learner learning?” We outline the validity argument and investigation relevant to this question, for a computer-based repeated assessment of comp...
Article
Rationale and Aim As medical schools reduce the hours of anatomy teaching, residents in anatomy-intensive residency programs like radiology must independently acquire the anatomy knowledge needed to achieve competency. The purpose of this study was to develop and evaluate a 4-week competency-based self-directed anatomy rotation for junior residents...
Article
Context The concept of entrustment has garnered significant attention in medical specialties, despite variability in supervision styles and entrustment decisions. There is a need to further study the enactment of supervision on inpatient wards to inform competency‐based assessment design. Methods Attending physicians, while supervising on clinical...
Article
Full-text available
IntroductionDirect observation of clinical skills is a key assessment strategy in competency-based medical education. The guidelines presented in this paper synthesize the literature on direct observation of clinical skills. The goal is to provide a practical list of Do’s, Don’ts and Don’t Knows about direct observation for supervisors who teach le...
Article
Context: Evidence strongly supports that direct observation is a valid and reliable assessment tool; support for its impact on learning is less compelling, and we know that some learners are ambivalent about being observed. However, learners' perceptions about the impact of direct observation on their learning and professional development remain u...
Article
Purpose: In-training evaluation reports (ITERs) constitute an integral component of medical student and postgraduate physician trainee (resident) assessment. ITER narrative comments have received less attention than the numeric scores. The authors sought both to determine what validity evidence informs the use of narrative comments from ITERs for...
Article
Full-text available
Background Simulation plays a vital role in health professions assessment. This review provides a primer on assessment validation for educators and education researchers. We focus on simulation-based assessment of health professionals, but the principles apply broadly to other assessment approaches and topics. Key principlesValidation refers to the...
Article
Context: Direct observation promises to strengthen both coaching and assessment, and calls for its increased use in medical training abound. Despite its apparent potential, the uptake of direct observation in medical training remains surprisingly limited outside the formal assessment setting. The limited uptake of observation raises questions abou...
Article
Context: We developed, implemented and evaluated an evidence-based programme of feedback designed to address limitations identified in the current literature. Objectives: We sought to advance understanding about how and why feedback processes might be more effective in clinical education. Methods: Three faculty members and nine first-year inte...
Article
Full-text available
Background: Simulation-based training is currently embedded in most health professions education curricula. Without evidence for how trainees think about their simulation-based learning, some training techniques may not support trainees' learning strategies. Objective: This study explored how residents think about and self-regulate learning duri...
Article
Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of j...
Article
Context: Progress tests, in which learners are repeatedly assessed on equivalent content at different times in their training and provided with feedback, would seem to lend themselves well to a competency-based framework, which requires more frequent formative assessments. The objective structured clinical examination (OSCE) progress test is a rel...
Article
Full-text available
In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected...
Article
Introduction: Clinical reasoning is foundational to medical practice, but the cognitive processes involved have been difficult to elucidate. Attention has been paid to cognitive biases and their role in diagnostic errors, but less attention has focused on the processes underlying successful diagnosis. We aimed to elucidate the factors associated wi...
Article
Objectives: Easily palpable splenomegaly can be identified on physical examination, but it is difficult to detect lesser degrees of splenomegaly. Rapid bedside assessment can be conducted with hand-carried ultrasound. We performed this study to determine whether medical residents could reliably assess spleen size using hand-carried ultrasound afte...
Article
ContextAssessment is central to medical education and the validation of assessments is vital to their use. Earlier validity frameworks suffer from a multiplicity of types of validity or failure to prioritise among sources of validity evidence. Kane's framework addresses both concerns by emphasising key inferences as the assessment progresses from a...
Article
ContextSelf-regulated learning (SRL) requires an active learner who has developed a set of processes for managing the achievement of learning goals. Simulation-based training is one context in which trainees can safely practise learning how to learn.Objectives The purpose of the present study was to evaluate, in the simulation-based training contex...
Article
Learning curves, which graphically show the relationship between learning effort and achievement, are common in published education research but are not often used in day-to-day educational activities. The purpose of this article is to describe the generation and analysis of learning curves and their applicability to health professions education. T...
Article
Full-text available
In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected...
Article
ContextThe relative advantages and disadvantages of checklists and global rating scales (GRSs) have long been debated. To compare the merits of these scale types, we conducted a systematic review of the validity evidence for checklists and GRSs in the context of simulation-based assessment of health professionals.Methods We conducted a systematic r...
Article
Full-text available
Purpose: To examine the evidence supporting the use of simulation-based assessments as surrogates for patient-related outcomes assessed in the workplace. Method: The authors systematically searched MEDLINE, EMBASE, Scopus, and key journals through February 26, 2013. They included original studies that assessed health professionals and trainees u...
Article
Clinical decisions should be based on the totality of the best evidence and not the results of individual studies. When clinicians apply the results of a systematic review or meta-analysis to patient care, they should start by evaluating the credibility of the methods of the systematic review, ie, the extent to which these methods have likely prote...
Article
Objectives: Physical examination can identify palpable splenomegaly easily, but evaluating lesser degrees of splenomegaly is problematic. Hand-carried ultrasound allows rapid bedside assessment of patients. We conducted this study to determine whether hand-carried ultrasound can reliably assess spleen size. Methods: Patients with varying degrees...
Article
Many education research studies employ small samples, which in turn lowers statistical power. We re-analyzed the results of a meta-analysis of simulation-based education to determine study power across a range of effect sizes, and the smallest effect that could be plausibly excluded. We systematically searched multiple databases through May 2011, a...
Article
This article provides an overview of cognitive load theory (CLT) and explores applications of CLT to health profession and nursing education research, particularly for multimedia and simulation-based applications. The article first reviews the 3 components of cognitive load: intrinsic, extraneous, and germane. It then discusses strategies for manip...
Article
In simulation-based health professions education, the concept of simulator fidelity is usually understood as the degree to which a simulator looks, feels, and acts like a human patient. Although this can be a useful guide in designing simulators, this definition emphasizes technological advances and physical resemblance over principles of education...
Article
Blinded assessments of technical skills using video-recordings may offer more objective assessments than direct observations. This study seeks to compare these two modalities. Two trained assessors independently assessed 18 central venous catheterization performances by direct observation and video-recorded assessments using two tools. Although sou...
Article
Introduction/Background: Simulation has been widely adopted in the teaching of procedural skills to both medical and surgical trainees. Simulation-based medical education (SBME) provides learners with the opportunity to develop and refine skills without jeopardizing patient safety. As such, simulation offers an ideal environment for trainees to eng...
Article
The optimal learner to simulator ratio for procedural skills training is not known. Research in motor learning suggests observational training in pairs, termed 'dyad training', may be as effective as directed self-regulated learning (DSRL). This study was conducted to compare the relative effectiveness and efficiency of dyad versus DSRL training of...
Article
Introduction/Background: Simulation-based assessments could serve as surrogates for workplace based assessments in medical education, decreasing the potential for patient harm and conserving resources by demonstrating learner competence in a safe training environment. While over reliance on surrogates should be avoided, once the necessary links are...
Article
Competency-based education requires individualization of instruction. Mastery learning, an instructional approach requiring learners to achieve a defined proficiency before proceeding to the next instructional objective, offers one approach to individualization. The authors sought to summarize the quantitative outcomes of mastery learning simulatio...
Article
Full-text available
Although feedback has been identified as a key instructional feature in simulation based medical education (SBME), we remain uncertain as to the magnitude of its effectiveness and the mechanisms by which it may be effective. We employed a meta-analysis and critical narrative synthesis to examine the effectiveness of feedback for SBME procedural ski...
Article
Full-text available
Ongoing transformations in health professions education underscore the need for valid and reliable assessment. The current standard for assessment validation requires evidence from five sources: content, response process, internal structure, relations with other variables, and consequences. However, researchers remain uncertain regarding the types...
Article
Purpose: To summarize the tool characteristics, sources of validity evidence, methodological quality, and reporting quality for studies of technology-enhanced simulation-based assessments for health professions learners. Method: The authors conducted a systematic review, searching MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, a...
Chapter
Assessment is an essential feature of the competency-based educational model because only by means of evaluation can we verify achievement of specified learning outcomes. This is especially important in the context of health professions education, where the competencies of interest impact the well-being of patients. Therefore, just as with planning...
Chapter
Simulation training for noninvasive cardiology has a rich history, beginning with heart sound simulators to the current era of multimedia simulations and full-scale cardiopulmonary simulators for learning cardiac physical examination. More recently, part-task trainers and mannequin-based simulators have been developed for training in transthoracic...
Article
Full-text available
Background: The quality of medical student and resident clinical evaluation reports submitted by rotation supervisors is a concern. The effectiveness of faculty development (FD) interventions in changing report quality is uncertain. Aims: This study assessed whether faculty could be trained to complete higher quality reports. Method: A 3-h int...
Article
To determine the comparative effectiveness of technology-enhanced simulation, we summarized the results of studies comparing technology-enhanced simulation training with nonsimulation instruction for health professions learners. We systematically searched databases including MEDLINE, Embase, and Scopus through May 2011 for relevant articles. Workin...
Article
OBJECTIVES The current review examines the effectiveness of simulation-based medical education (SBME) for training health professionals in cardiac physical examination and examines the relative effectiveness of key instructional design features. METHODS Data sources included a comprehensive, systematic search of MEDLINE, EMBASE, CINAHL, PsychINFO,...
Article
Full-text available
Background: Although technology-enhanced simulation is increasingly used in health professions education, features of effective simulation-based instructional design remain uncertain. Aims: Evaluate the effectiveness of instructional design features through a systematic review of studies comparing different simulation-based interventions. Methods:...
Article
Simulation training offers opportunities for unsupervised, self-regulated learning, yet little evidence is available to indicate the efficacy of this approach in the learning of procedural skills. We evaluated the effectiveness of directed self-regulated learning (DSRL) and instructor-regulated learning (IRL), respectively, for teaching lumbar punc...
Article
Full-text available
Use of dual-processing has been widely touted as a strategy to reduce diagnostic error in clinical medicine. However, this strategy has not been tested among medical trainees with complex diagnostic problems. We sought to determine whether dual-processing instruction could reduce diagnostic error across a spectrum of experience with trainees undert...
Article
The uncertainty around number needed to treat (NNT) is often represented through a confidence interval (CI). However, it is not clear how the CI can help inform treatment decisions. We developed decision-theoretic measures of uncertainty for the NNT. We build our argument on the basis that a risk-neutral decision maker should always choose the trea...
Article
Most efforts to develop reliable evaluations of clinical competence have been oriented toward deconstructing the requisite competencies into separate scales. However, many are questioning the value of this approach on theoretical and empirical bases. This study uses "standardized narratives" to explore a different approach to assessing resident per...
Article
Full-text available
Although technology-enhanced simulation has widespread appeal, its effectiveness remains uncertain. A comprehensive synthesis of evidence may inform the use of simulation in health professions education. To summarize the outcomes of technology-enhanced simulation training for health professions learners in comparison with no intervention. Systemati...
Article
The use of checklists is recommended for the assessment of competency in central venous catheterization (CVC) insertion. To explore the use of a global rating scale in the assessment of CVC skills, this study seeks to compare its use with two checklists, within the context of a formative examination using simulation. Video-recorded performances of...
Article
As the use of simulation-based assessment expands for healthcare workers, there is a growing need for research to quantify the psychometric properties of the associated process and outcome measures.