Rose Hatala’s research while affiliated with Universidade Braz Cubas and other places

Indirect supervision is essential for granting autonomy to learners. Sometimes referred to as leaving the learner “unsupervised,” there is growing recognition that learners and supervisors engage in clinical support through ongoing interactions, albeit at a distance. To understand how the complementary activities of asking for and providing clinical support at a distance come together as indirect supervision by identifying the natural language used between learners and supervisors. A collective case study of 15 attending physician-senior medical resident dyads from 2018 to 2023. Each case consisted of 2 full days of ethnographic observation of the resident on an internal medicine clinical teaching unit in one of three metropolitan hospitals; 2 end of day interviews with each dyad member; and a third interview with the attending after the 2-week rotation. Fifteen internal medicine residents (PGY-2 and PGY-3) who were scheduled to work for the first time with 15 attending physicians. Data collection was iterative with deductive and inductive analysis to identify patterns of communication. The language of “checking,” such as checkpoints, checking on, and checking in, was central to communications within dyads. Indirect supervision included attendings using scheduled checkpoints and backstage oversight activities to check on the senior resident’s patient care while expecting residents to access their support, as needed, by checking-in with them. Residents checked in with their attending to relay patient information updates, ask questions, and hint at needing their plans doublechecked; these communications had similar formats making them difficult to distinguish but functioned to preserve resident independence while accessing clinical support. Indirect supervision creates clinical support through ongoing communication between learners and supervisors at a distance. It is a collaborative process for mutual reassurance that safe patient care is being provided and that support is available when needed.

In this editorial the editors consider the ideals and realities of high and low stakes assessments in clinical workplaces, the impact of these assessments on clinical workplace learning, and the clash between authenticity in assessment and authenticity in learning.

Introduction Competence committees (CCs) centre their work around documentation of trainees' performance; undocumented contributions (i.e. informal, unrecorded material like personal judgements, experiential anecdotes and contextual information) evoke suspicion even though they may play a role in decision making. This qualitative multiple case study incorporates insights from a social practice perspective on writing to examine the use of undocumented contributions by the CCs of two large post‐graduate training programmes, one in a more procedural (MP) speciality and the other in a less procedural (LP) one. Methods Data were collected via observations of meetings and semi‐structured interviews with CC members. In the analysis, conversations were organised into triptychs of lead‐up, undocumented contribution(s), and follow‐up. We then created thick descriptions around the undocumented contributions, drawing on conversational context and interview data to assign possible motivations and significance. Results We found no instances in which undocumented contributions superseded the contents of a trainee's file or stood in for missing documentation. The number of undocumented contributions varied between the MP CC (six instances over two meetings) and the LP CC (22 instances over three meetings). MP CC discussions emphasised Entrustable Professional Activity (EPA) observations, whereas LP CC members paid more attention to narrative data. The divergent orientations of the CCs—adding an ‘advis[ing]/guid[ing]’ role versus focusing simply on evaluation—offers the most compelling explanation. In lead‐ups, undocumented contributions were prompted by missing and flawed documentation, conflicting evidence and documentation at odds with members' perceptions. Recognising other ‘red flags’ in documentation often required professional experience. In follow‐ups, purposes served by undocumented contributions varied with context and were difficult to generalise; we, therefore, provide deeper analysis of two vignettes to illustrate. Conclusions Our data suggest undocumented contributions often serve best efforts to ground decisions in documentation. We would encourage CC practices and policies be rooted in more nuanced approaches to documentation.

The authors suggest reducing the use of WBA where it is not fit for purpose and developing locally sustainable and defensible programs of assessment as steps towards unlocking the value of CBME.

Longitudinal academic advising (AA) and coaching programs are increasingly implemented in competency based medical education (CBME) to help residents reflect and act on the voluminous assessment data they receive. Documents created by residents for purposes of reflection are often used for a second, summative purpose—to help competence committees make decisions—which may be problematic. Using inductive, thematic analysis we analyzed written comments generated by 21 resident-AA dyads in one large internal medicine program who met over a 2 year period to determine what residents write when asked to reflect, how this aligns with what the AAs report, and what changes occur over time (total 109 resident self-reflections and 105 AA reports). Residents commented more on their developing autonomy, progress and improvement than AAs, who commented far more on performance measures. Over time, residents’ writing shifted away from intrinsic roles, patient care and improvement towards what AAs focused on, including getting EPAs (entrustable professional activities), studying and exams. For EPAs, the emphasis was on getting sufficient numbers rather than reflecting on what residents were learning. Our findings challenge the practice of dual-purposing documents, by questioning the blurring of formative and summative intent, the structure of forms and their multiple conflicting purposes, and assumptions about the advising relationship over time. Our study suggests a need to re-evaluate how reflective documents are used in CBME programs. Further research should explore whether and how documentation can best be used to support resident growth and development.

Purpose: Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician's decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example. Method: The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from "definitely STEMI" to "definitely pericarditis," including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale ("definitely STEMI" to "definitely pericarditis"). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review. Results: Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories. Conclusions: The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.

Background: Longitudinal coaching in residency programs is becoming commonplace and requires iterative and collaborative discussions between coach and resident, with the shared development of goals. However, little is known about how goal development unfolds within coaching conversations over time and the effects these conversations have. We therefore built on current coaching theory by analysing goal development dialogues within resident and faculty coaching relationships. Methods: This was a qualitative study using interpretive description methodology. Eight internal medicine coach-resident dyads consented to audiotaping coaching meetings over a one-year period. Transcripts from meetings and individual exit interviews were analyzed thematically using goal co-construction as a sensitizing concept. Results: Two themes were developed: (1) the content of goals discussed in coaching meetings focused on how to be a resident, with little discussion around challenges in direct patient care, and (2) co-construction mainly occurred in how to meet goals, rather than in prioritizing goals or co-constructing new goals. Discussion: In analysing goal development in the coach-resident relationships, conversations focused mainly around how to manage as a resident rather than how to improve direct patient care. This may be because academic coaching provides space separate from clinical work to focus on the stage specific professional identity development of a resident. Going forward, focus should be on how to optimize longitudinal coaching conversations to ensure co-regulation and reflection on both clinical competencies and professional identity formation.

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals’ assessment were included. We organized and synthesized the validity evidence according to Kane’s framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

Introduction: Postgraduate competency-based medical education has been implemented with programmatic assessment that relies on entrustment-based ratings. Yet, in less procedurally oriented specialties such as internal medicine, the relationship between entrustment and supervision remains unclear. We undertook the current study to address how internal medicine supervisors conceptualise entrusting senior medical residents while supervising them on the acute care wards. Methods: Guided by constructivist grounded theory, we interviewed 19 physicians who regularly supervised senior internal medicine residents on inpatient wards at three Canadian universities. We developed a theoretical model through iterative cycles of data collection and analysis using a constant comparative process. Results: On the internal medicine ward, the senior resident role is viewed as a fundamentally managerial and rudimentary version of the supervisor's role. Supervisors come to trust their residents in the senior role through an early 'hands-on' period of assessment followed by a gradual withdrawal of support to promote independence. When considering entrustment, supervisors focused on entrusting a particular scope of the senior resident role as opposed to entrustment of individual tasks. Irrespective of the scope of the role that was entrusted, supervisors at times stepped in and stepped back to support specific tasks. Conclusion: Supervisors' stepping in and stepping back to support individual tasks on the acute care ward has an inconsistent relationship to their entrustment of the resident with a particular scope of the senior resident role. In this context, entrustment-based assessment would need to capture more of the holistic perspective of the supervisor's entrustment of the senior resident role. Understanding the dance of supervision, from relatively static overall support of the resident in their role, to fluidly stepping in and out for specific patient care tasks, allows us insight into the affordances of the supervisory relationship and how it may be leveraged for assessment.