Shiphra Ginsburg

• MD, MEd, PhD
• Professor at University of Toronto

Written comments on residents' evaluations can be useful, yet the literature suggests that the language used by assessors is often vague and indirect. The branch of linguistics called pragmatics argues that much of our day to day language is not meant to be interpreted literally. Within pragmatics, the theory of 'politeness' suggests that non-liter...

Quantitative scores fail to capture all important features of learner performance. This awareness has led to increased use of qualitative data when assessing health professionals. Yet the use of qualitative assessments is hampered by incomplete understanding of their role in forming judgments, and lack of consensus in how to appraise the rigor of j...

Narrative comments are used routinely in many forms of rater-based assessment. Interpretation can be difficult as a result of idiosyncratic writing styles and disconnects between literal and intended meanings. Our purpose was to explore how faculty attendings interpret and make sense of the narrative comments on residents' in-training evaluation re...

Implementing competency based medical education (CBME) has generated enormous amounts of assessment data. To help residents synthesize and use these data, some programs have appointed academic advisors (AA) to ‘coach over time’. This study explored how resident and faculty AA dyads perceived their relationship developing and evolving, and the exten...

Importance Variation in residency case exposure affects resident learning and readiness for future practice. Accurate reporting of case exposure for internal medicine (IM) residents is challenging because feasible and reliable methods for linking patient care to residents are lacking. Objective To develop an integrated education-clinical database...

Background/Objective In implementing competence‐based medical education (CBME), some Canadian residency programmes recruit clinicians to function as Academic Advisors (AAs). AAs are expected to help monitor residents' progress, coach them longitudinally, and serve as sources of co‐regulated learning (Co‐RL) to support their developing self‐regulate...

Purpose Written assessment comments are needed to archive feedback and inform decisions. Regrettably, comments are often impoverished, leaving performance-relevant information undocumented. Research has focused on content and supervisor’s ability and motivation to write it but has not sufficiently examined how well the undocumented information lend...

Purpose Feedback seeking is an expected learner competency. Motivations to seek feedback are well explored, but we know little about how supervisors perceive such requests for feedback. These perceptions matter because how supervisors judge requests can affect the feedback they give. This study explores how supervisors perceive and attribute motiva...

Purpose The MeToo movement forced a social reckoning, spurring women in medicine to engage in the #MeTooMedicine online discourse. Given the risks of reporting sexual violence, discrimination, or harassment, it is important to understand how women in medicine use platforms like Twitter to publicly discuss their experiences. With such knowledge, the...

Longitudinal academic advising (AA) and coaching programs are increasingly implemented in competency based medical education (CBME) to help residents reflect and act on the voluminous assessment data they receive. Documents created by residents for purposes of reflection are often used for a second, summative purpose—to help competence committees m...

Precision education (PE) leverages longitudinal data and analytics to tailor educational interventions to improve patient, learner, and system-level outcomes. At present, few programs in medical education can accomplish this goal as they must develop new data streams transformed by analytics to drive trainee learning and program improvement. Other...

Background: The cause of physician burnout is multifactorial. Health care systems pressures, excessive workloads, fatigue, poor self-care, administrative burdens, work hours, technological advancements, and work-home life conflicts, are all prominent themes throughout the literature. To date, little is known about whether, and to what extent, stre...

Introduction: Current orthodoxy states that feedback should be timely and face-to-face, yet the optimal timing and mode of delivery for feedback is unclear. We explored what "optimal timing" means from residents' points of view as feedback providers and receivers, to ultimately inform strategies to optimize feedback in training. Methods: As near...

Background Internal Medicine (IM) residents are required to perform bedside procedures for diagnostic and therapeutic purposes. Residents’ experiences with procedures vary widely, for unclear reasons. Objective To explore IM residents’ experiences with performing bedside procedures and to identify barriers and facilitators to obtaining sufficient...

Background Physician-patient communication training is a vital component of medical education, yet physicians do not always achieve the communication expertise expected of them. Despite extensive literature on the efficacy of various training interventions, little is known about how residents believe they learn to communicate. Objective To underst...

The evaluation of clinical teachers’ performance has long been a subject of research and debate, yet teaching evaluations (TEs) by students remain problematic. Despite their intuitive appeal, there is little evidence that TEs are associated with students’ learning in the classroom or clinical setting. TEs are also subject to many forms of bias and...

Introduction: Unwarranted variation in patient care among physicians is associated with negative patient outcomes and increased healthcare costs. Care variation likely also exists for resident physicians. Despite the global movement towards outcomes-based and competency-based medical education, current assessment strategies in residency do not rou...

Background: Peer review aims to provide meaningful feedback to research authors so that they may improve their work, and yet it constitutes a particularly challenging context for the exchange of feedback. We explore how research authors navigate the process of interpreting and responding to peer review feedback, in order to elaborate how feedback...

Purpose: Entrustable Professional Activities (EPA) assessments are intended to facilitate meaningful, low-stakes coaching and feedback, partly through the provision of written comments. We sought to explore EPA assessment comments provided to internal medicine (IM) residents for evidence of feedback and coaching language as well as politeness. Me...

Introduction: Postgraduate competency-based medical education has been implemented with programmatic assessment that relies on entrustment-based ratings. Yet, in less procedurally oriented specialties such as internal medicine, the relationship between entrustment and supervision remains unclear. We undertook the current study to address how inter...

Assessment of clinical teachers by learners is problematic. Construct-irrelevant factors influence ratings, and women teachers often receive lower ratings than men. However, most studies focus only on numeric scores. Therefore, the authors analyzed written comments on 4032 teacher assessments, representing 282 women and 448 men teachers in one Depa...

Background: Research has acknowledged the value of patients as essential stakeholders in medical education, yet educators have not adequately incorporated patients' perspectives into medical students' developing professionalism. Our purpose was to explore patients' perceptions of professional behaviour in medical students as a first step to consid...

A taskforce established by Medical Education asks readers to engage in discussion about how the journal and field can do better to ensure that health professional education publishing is inclusive of diverse knowledge and perspectives.

Purpose: Entrustment is central to assessment in competency-based medical education (CBME). To date, little research has addressed how clinical supervisors conceptualize entrustment, including factors they consider in making entrustment decisions. The aim of this study was to characterize supervisors' decision making related to procedural entrustm...

Purpose: With the introduction of competency-based medical education (CBME), senior residents have taken on a new, formalized role of completing assessments of their junior colleagues. However, no prior studies have explored the role of near-peer assessment within the context of entrustable professional activities (EPAs) and CBME. This study explo...

Background: Residents frequently lead goals of care (GoC) conversations with patients and families to explore patient values and preferences and to establish patient-centered care plans. However, previous work has shown that the hidden curriculum may promote physician-driven agendas and poor communication in these discussions. We previously develop...

Purpose: Multiple studies demonstrate the assessment of residents differs by gender, yet little is known about how these differences are experienced by women and men. The authors sought to understand whether the experience of being assessed and receiving feedback differs between men and women internal medicine (IM) residents and how women respond...

Introduction Mortality and morbidity are high for older adults after hip fracture (HF), but patients and surrogate decision makers (SDMs) are typically unaware of the poor prognosis. We developed a novel educational tool, My Hip Fracture (My-HF), to provide patients and SDMs of patients hospitalized with acute HF individualized estimates of their p...

The current discourse on competency-based medical education (CBME) is confounded by a lack of agreement on definitions and philosophical assumptions. This phenomenon impacts curriculum implementation, program evaluation and disrupts dialogue with the education community. The purpose of this study is to explore how Canadian key opinion leaders descr...

Background: Gender disparities in medical education are increasingly demonstrated, including in trainee assessment. Objective: This study aimed to evaluate whether gender differences exist in trainees’ evaluation during intensive care unit (ICU) rotations, which has not been previously studied. Methods: We reviewed the in-training evaluation report...

The adoption of entrustment ratings in medical education is based on a seemingly simple premise: to align workplace-based supervision with resident assessment. Yet it has been difficult to operationalize this concept. Entrustment rating forms combine numeric scales with comments and are embedded in a programmatic assessment framework, which encoura...

Introduction: Reports have forecast significant changes ahead for Continuing Professional Development (CPD). With new models on the horizon, CPD developers will require additional knowledge and training. Our objective was to learn how CPD leaders and developers came into their roles, if they are engaged in scholarship and the challenges and opport...

Background Learner‐centred medical education relies on learner agency. While attractive in principle, the actual exercise of agency is a complicated process, potentially constrained by social norms and cultural expectations. In this study, we explored what it means to be an agentic learner in medicine, and how individuals experience and harness age...

Background: There has been growing acknowledgement that undergraduate medical education (UME) must play a formal role in instructing future physicians on the promises and limitations of artificial intelligence (AI), as these tools are integrated into medical practice. Methods: We conducted an exploratory survey of medical students knowledge of AI,...

Background Gender bias is thought to exist in the assessment of clinical teachers, yet its extent in different specialties is not well-documented nor has it been studied at the individual-dyadic level. Objective The authors sought to determine whether gender bias exists in residents' assessments of faculty teaching in 3 clinical departments, and i...

Introduction Capitalizing on direct workplace observations of residents by interprofessional team members might be an effective strategy to promote formative feedback in postgraduate medical education. To better understand how interprofessional feedback is conceived, delivered, received, and used, we explored both feedback provider and receiver per...

Purpose: Learner handover is the sharing of information about learners between faculty supervisors. Learner handover can support longitudinal assessment in rotation-based systems, but there are concerns that the practice could bias future assessments or stigmatize struggling learners. Because successful implementation relies on an understanding of...

Purpose: Models of daytime and nighttime on-call responsibilities for residents vary across internal medicine training programs, but there are few data regarding residents' perceptions of their on-call experiences. The authors sought to understand what residents perceive as the benefits and detriments of 24-hour, in-house call, a perspective instr...

Purpose: As educators have implemented competency-based medical education (CBME) as a framework for training and assessment, they have made decisions based on available evidence and on the medical education community’s assumptions about CBME. This critical narrative review aimed to collect, synthesize, and judge the existing evidence underpinning...

Background: Older adults with functional impairment are cared for by physiatrists in rehabilitation, but physiatrist training in geriatric-related competencies remains suboptimal. To develop a geriatric rehabilitation (GR) curriculum and explore opportunities for improvement, a needs assessment of stakeholders was conducted to understand physical...

Purpose: Written comments are gaining traction as robust sources of assessment data. Compared to the structure of numeric scales, what faculty choose to write is ad hoc, leading to idiosyncratic differences in what is recorded. This study offers exploration of what aspects of writing styles are determined by the faculty offering comment and what a...

Purpose: As key participants in the assessment dyad, residents must be engaged with the process. However, residents' experiences with competency-based medical education (CBME), and specifically with entrustable professional activity (EPA)-based assessments, have not been well studied. The authors explored junior residents' perceptions regarding th...

Background Competency-based medical education (CBME), an educational paradigm that prioritizes development of measurable skills over time in training, is currently being instituted across North American residency training programs. A fundamental goal of CBME is entrustment – the process whereby supervisors come to trust trainees to perform specific...

Purpose: Written comments are increasingly valued for assessment; however, a culture of politeness and the conflation of assessment with feedback lead to ambiguity. Interpretation requires reading between the lines, which is untenable with large volumes of qualitative data. For computer analytics to help with interpreting comments, the factors inf...

This AMEE guide provides a framework and practical strategies for teachers, learners and institutions to promote meaningful feedback conversations that emphasise performance improvement and professional growth. Recommended strategies are based on recent feedback research and literature, which emphasise the sociocultural nature of these complex inte...

Purpose: This systematic review sought to summarize published professionalism curricula in postgraduate medical education (PGME) and identify best practices for teaching professionalism. Method: Three databases (MEDLINE, Embase, ERIC) were searched for articles published from 1980 through September 7, 2017. English-language articles were include...

Purpose: A key unit of assessment in competency-based medical education (CBME) is the entrustable professional activity. The variations in how entrustment is perceived and enacted across specialties are not well understood. This study aimed to develop a thorough understanding of the process, concept, and language of entrustment as it pertains to i...

Purpose: Newer definitions of feedback emphasize learner engagement throughout the conversation, yet teacher and learner perceptions of each other's behaviors during feedback exchanges have been less well studied. This study explored perceptions of residents and faculty regarding effective behaviors and strategies during feedback conversations and...

Background Technology such as wearable technology and self-management applications could improve the care of patients with chronic obstructive pulmonary disease (COPD) by real-time continuous monitoring, early detection of COPD and improved self-management. However, patients have not been willing to use technology when it is too difficult to use, i...

The implementation of Entrustable Professional Activities has led to the simultaneous development of assessment based on a supervisor’s entrustment of a learner to perform these activities without supervision. While entrustment may be intuitive when we consider the direct observation of a procedural task, the current implementation of rating scales...

Purpose: Medical schools are expected to promote compassionate care among learners. Assessment is a key way to communicate values to learners but can create a hidden curriculum. Assessing compassionate care is challenging; however, not assessing it can communicate to students that such care is not valued. The purpose of this study was to explore h...

Feedback is defined as a regulatory mechanism where the effect of an action is fed back to modify and improve future action. In medical education, newer conceptualizations of feedback place the learner at the center of the feedback loop and emphasize learner engagement in the entire process. But, learners reject feedback if they doubt its credibili...

Background: In 2009, an International Working Group (IWG) on the Assessment of Professionalism began collaborating and published recommendations in 2011. Nearly a decade later the IWG reconvened to take stock of the state of practice and research in professionalism and the impact of the 2011 report. Method: A bibliometric study identified all publi...

Background: In 2009, an International Working Group (IWG) on the Assessment of Professionalism began collaborating and published recommendations in 2011. Nearly a decade later the IWG reconvened to take stock of the state of practice and research in professionalism and the impact of the 2011 report. Method: A bibliometric study identified all publi...

Phenomenon: Clerkship is a challenging transition during which medical students must learn to navigate the responsibilities of medical school and clinical medicine. We explored how clerks understand their roles as both medical learners and developing professionals and some of the tensionss that arise therein. Understanding how the clinical learning...

BACKGROUND Podcasts are increasingly produced for medical education, yet their use and perceived impact in informal educational settings is under-studied. OBJECTIVE We explored how and why physicians and medical learners listen to The Rounds Table (TRT), a medical podcast, and the perceived impact on learning and practice. METHODS Web-based podca...

Background: Although podcasts are increasingly being produced for medical education, their use and perceived impact in informal educational settings are understudied. Objective: This study aimed to explore how and why physicians and medical learners listen to The Rounds Table (TRT), a medical podcast, as well as to determine the podcast's percei...

Phenomenon: Medical professionalism is a complex construct, based in social and cultural influences, yet little research has been done to show how culture influences the behaviors and attitudes of medical students. We presented Italian students with the same professional dilemmas used in a previous Canadian and Taiwanese cross-cultural study to loo...

Purpose: Competence is bound to context, yet seldom is environment explicitly considered in work-based assessments. This study explored faculty and residents' perspectives of the environment during internal medicine clinical teaching unit (CTU) rotations, the extent that each group accounts for environmental factors in assessments, and relationshi...

Context: Models of sound assessment practices increasingly emphasise assessment's formative role. As a result, assessment must not only support sound judgements about learner competence, but also generate meaningful feedback to guide learning. Reconciling the tension between assessment's focus on judgement and decision making and feedback's focus...

Purpose: Although the field of medical education research is growing and residents are increasingly recruited to participate as subjects in research studies, little is known about their experiences. The goal of this study was to explore the experiences and perceptions of residents who are study participants in medical education research. Method:...

Purpose: A rise in incivility has been documented in medicine, with implications for patient care, organizational effectiveness, and costs. This study explored organizational factors that may contribute to incivility at one academic medical center and potential systems-level solutions to combat it. Method: The authors completed semistructured in...

Purpose: Medical practice is uncertain and complex. Consequently, even outstanding performers will inevitably experience moments of underperformance and failure. Coping relies on insight and resilience. However, how physicians develop and use these skills to navigate struggle remains underexplored. A better understanding may reveal strategies to s...

Purpose: Publishing in peer-reviewed journals is essential for medical education researchers. Competition remains fierce for top journals and authors are advised to consider impact factor (IF), audience, and alignment of focus. However, little is known about how authors balance these factors when making submission decisions. The authors aimed to e...

Feedback in medical education has traditionally showcased techniques and skills of giving feedback, and models used in staff development have focused on feedback providers (teachers) not receivers (learners). More recent definitions have questioned this approach, arguing that the impact of feedback lies in learner acceptance and assimilation of fee...

Introduction: In the growing field of medical education research, participant recruitment can be challenging. Incentives, either tangible or intangible, may be offered to encourage participation. This study aimed to understand these incentives and explore the relationship between study quality and incentives in medical education research. Methods:...

Purpose: Mistakes are ubiquitous in medicine; when confronted by error, physicians may experience anxiety, guilt, and self-doubt. Feedback may be useful for navigating these feelings, but only if it matches a physician's self-assessment; self-doubt and the imposter syndrome are examples of inaccurate self-assessments that may affect receptivity to...

Background: Over the last two decades, competency-based frameworks have been internationally adopted as the primary educational approach in medicine. Yet competency-based medical education (CBME) remains contested in the academic literature. We look broadly at the nature of this debate to explore how it may shape scholars' understanding of CBME, a...

Dr Patel is an internist who recently joined a large multidisciplinary clinic. She often refers her patients to 1 of the 6 cardiologists who work there. She has noticed that when her patients are seen by Dr Wells, they almost always undergo stenting procedures, whereas when her patients are seen by the other cardiologists, a larger variety of treat...

Dr Bernard is a 70-year-old endocrinologist whose passion in life has been the practice of medicine and the care of patients with diabetes. He noticed that he was getting fewer referrals for new patients from other physicians in his group. On 2 occasions, other clinicians made passing comments to him about retirement. Although he has had a busy and...

Purpose: In-training evaluation reports (ITERs) are ubiquitous in internal medicine (IM) residency. Written comments can provide a rich data source, yet are often overlooked. This study determined the reliability of using variable amounts of commentary to discriminate between residents. Method: ITER comments from two cohorts of PGY-1s in IM at t...

Context: Interest is growing in the use of qualitative data for assessment. Written comments on residents' in-training evaluation reports (ITERs) can be reliably rank-ordered by faculty attendings, who are adept at interpreting these narratives. However, if residents do not interpret assessment comments in the same way, a valuable educational oppo...

Dr Facey, an internal medicine faculty member, is a well-known physician-scientist and respected leader in his scientific field of investigation. Dr Facey’s teaching style includes questioning students on material that is beyond their educational level, believing that this style motivates them to learn. He sometimes uses a sarcastic tone and says t...

Purpose: In-training evaluation reports (ITERs) constitute an integral component of medical student and postgraduate physician trainee (resident) assessment. ITER narrative comments have received less attention than the numeric scores. The authors sought both to determine what validity evidence informs the use of narrative comments from ITERs for...

Dr Silver works as an obstetrician-gynecologist in a small community. A new colleague, Dr King, joined the practice about 6 months ago and now comes to Dr Silver for help. Dr King is feeling a bit depressed, missing her old friends and former practice, and mentions that when she felt like this 5 years ago, a selective serotonin reuptake inhibitor (...

Despite multifaceted attempts to "protect the public," including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critica...

This JAMA Professionalism article presents the case of a medical error and the options for handling the situation, including communicating with the patients. Section Editor Edward H. Livingston, MD, Deputy Editor, JAMA. Article Information Corresponding Author Wendy Levinson, MD, Department of Medicine, St Michael’s Hospital, University of Toront...

Theory: Assessment of clinical competence is a complex cognitive task with many mental demands often imposed on raters unintentionally. We were interested in whether this burden might contribute to well-described limitations in assessment judgments. In this study we examine the effect on indicators of rating quality of asking raters to (a) conside...

Introduction: Context shapes behaviours yet is seldom considered when assessing competence. Our objective was to explore attending physicians' and trainees' perceptions of the Internal Medicine Clinical Teaching Unit (CTU) environment and how they thought contextual factors affected their performance. Method: 29 individuals recently completing C...

ContextAssessment is central to medical education and the validation of assessments is vital to their use. Earlier validity frameworks suffer from a multiplicity of types of validity or failure to prioritise among sources of validity evidence. Kane's framework addresses both concerns by emphasising key inferences as the assessment progresses from a...

Understanding medical professionalism and its evaluation is essential to ensuring that physicians graduate with the requisite knowledge and skills in this domain. It is important to consider the context in which behaviours occur, along with tensions between competing values and the individual’s approach to resolving such conflicts. However, too muc...

Introduction: Physicians do not always agree on "rules" governing behavior in professionally challenging situations. Little is known about contextual factors that influence physician approaches in these situations. We explored the individual-, social-, and societal-level factors that physicians consider when responding to 2 common professional dil...

To develop a deeper understanding of the complexity of physicians' decision making when faced with professional challenges. Using constructivist grounded theory, the authors conducted a secondary analysis of transcripts from focus groups with 40 internists in 2011. Participants responded to scripted professional challenge scenarios. The authors ana...

Corresponding Author: Shiphra Ginsburg, MD, MEd, FRCPC, Wilson Centre for Research in Education, 600 University Ave, Ste 433, Toronto, ON M5G 1X5, Canada (shiphra.ginsburg@utoronto.ca). Author Contributions: Dr Walsh had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data a...

Although scores on in-training evaluation reports (ITERs) are often criticized for poor reliability and validity, ITER comments may yield valuable information. The authors assessed across-rotation reliability of ITER scores in one internal medicine program, ability of ITER scores and comments to predict postgraduate year three (PGY3) performance, a...