ArticleLiterature Review

“Directed” self‐assessment: Practice and feedback within a social context

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Introduction: Accurate self-assessment appears to be difficult and, some would propose, even impossible. Recent reviews suggest that peer assessment may be more accurate and that multisource feedback (MSF) may inform self-assessment. We had conducted a series of studies of family physicians in an MSF program including assessments from patients, medical colleagues, and coworkers and self-assessment. Using this body of research, this article explores self-assessment within the social context of multisource feedback and investigates the influence of feedback from peers and others upon self-assessment. Methods: This is a review article in which we synthesized findings of the series of studies with respect to self-assessment, used conclusions to propose a model for self-assessment within a social context, and suggest practical and research implications. Results: Physicians compared peers' and others' assessment feedback with global self-perceptions of performance. Negative feedback, especially from medical colleagues, that was inconsistent with self-perceptions was not readily reconciled with self-assessments. Multiple internal and environmental factors influenced reconciliation and assimilation of negative feedback. Reflection upon feedback and self-perceptions appeared to be instrumental to reconciliation, and reflection could be facilitated. Discussion: We propose a model of “directed” self-assessment to facilitate the integration of external feedback, especially negative feedback, with self-perceptions and enable its use for practice improvement. Implications for education and research include increasing understanding of ways physicians assimilate external feedback and of the role of educators as facilitators of “directed” self-assessment and self-learning to assist physicians in integrating external feedback.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Eva et al. (2012) reported that feedback was more likely utilized and learning goals adjusted when the recipient received criticism that was aligned with their self-concept. Similarly, Sargeant et al. (2008) found that negative emotions resulted in the physician's inability or unwillingness to assimilate the feedback into their learning. To explore the participant's overall reaction to feedback and specifically their reaction to the encounters that may have resulted in an emotional response, the following prompts were developed: (a) Think back to a memorable feedback encounter. ...
... That might, in part, explain why those considered "high-performing" might utilize feedback differently than "low-performers." Sargeant et al. (2008) noted that those in a high performing group of physicians "appeared intentional, self-aware, self-directed and reflective in their learning." The prompt, "how did you use the feedback for learning and practice change?" was used to examine how well participants made and implemented a plan for learning, and if they described adaptive learning traits. ...
... Surprise and confusion can increase engagement and result in positive learning outcomes (Linnenbrink-Garcia et al. 2016), but the individuals who expressed surprise in this study also described having difficulty learning in the clinical setting. Surprise, and the subsequent negative emotional response it generated, was akin to the response to disconfirming feedback reported elsewhere (Eva et al. 2012;Sargeant et al. 2008). ...
Full-text available
Article
Educators have long noticed differences in how students receive and use feedback. Despite the development of best practice guidelines, some learners in the health professions still struggle to incorporate corrective feedback. To date, little research has been done to examine learner characteristics and how those traits might explain differences in feedback-related behavior. A qualitative study using a constructivist, grounded theory approach was conducted to examine the behaviors and learner characteristics that contribute to successful use of feedback. Medical and physician assistant students in their clinical years at one academic health center were interviewed to determine how they used feedback to learn. An overarching theory was developed to explain the process used by students who successfully used feedback. Participants needed to be ready, willing, and able to learn from feedback. Readiness required the learner to hear the corrective feedback without negative emotion and understand how to use the criticism. Willingness required a valuing process that resulted in meaning making. It also required motivation and a growth mindset. Able learners needed to utilize strategic learning behaviors to incorporate feedback into practice change. Barriers to feedback use included emotional interference, inability to create a positive narrative around the feedback encounter and understand it as part of a larger context, and the inability to utilize adaptive learning strategies. The paper considers important education theories, such as Self-regulated Learning and Achievement Goal Theory—theories that should be incorporated into future research.
... • Social interaction [1,[15][16][17] • Problem-based learning [18] • Point of care learning [19,20] • Addressing appropriate learning objectives [21] • Guided self-assessment [22][23][24] • Reflection [15,24] • Blended learning [25][26][27] • Use of technology [28] • Familiar content [28] • Inaccurate needs assessment [29,30] • Informal 'scanning' approaches [31] • Inexperience of learners [32] • Age of learners [31,32] • Unguided self-assessment [1] • Lack of social interaction [31] • Unstructured reflection or reflection on aggregate performance [24,33] • difficulties in trusting that information/content is accurate, peer reviewed, and/or credible. Several studies have identified strategies for successful outcomes associated with SDL: dedicating time, creating accountability, and helping with goal generation through mentorship or relying more on external assessments than on self-assessment [35][36][37]. ...
... Additionally, SDL and lifelong learning rely heavily on accurate self-assessment, but several studies have found inaccuracies in physicians' self-assessments of their knowledge and skills. Physicians with lowest external performance and with less experience tend to overestimate their competence [21,[52][53][54][55]. Some suggest the contradictory evidence surrounding SDL may be caused by inconsistent definitions and interpretations of SDL, particularly as applied to CPD programmes in medicine [8,56,57]. ...
... The creation of a learning plan based on a response to patient and population health needs is envisioned to require an integration of self-appraised needs with external measures of performance and health outcomes. Models of 'informed self-assessment' as described by Sargeant et al. [74], coupled with evidence-informed feedback strategies [21,[74][75][76], are viewed as important in 'making sense' of external sources of data particularly when they conflict with the physicians' views of their performance. ...
Chapter
Physicians' professional development and lifelong learning model has shifted to a broader model of Continuing Professional Development (CPD), which encompasses diverse activities, addresses diverse aspects of physicians' competency, and includes diverse professionals, highlighting the importance of team‐based interprofessional education. Recently, the emphasis of physicians' professional development has been shifting toward the broader concept of CPD and away from the narrower traditional view where it was perceived as a didactic and clinically oriented activity. This chapter discusses four different elements that are considered pertinent, in our views, to the current model of CPD for physicians and other health care professionals: self‐directed learning; the relationship between maintenance of certification and professional development; the competencies across the medical education continuum; and team‐based education in CPD. Self‐directed learning is a learning process that is considered to be one of the most appropriate strategies in CPD for physicians.
... Sociocultural factors that influence feedback include: teacher characteristics, teacher-learner relationships, observation of performance, learner self-efficacy, autonomy, feedback seeking and acceptance, and perceived credibility of feedback data (Sargeant et al. 2007(Sargeant et al. , 2008aDelva et al. 2011;Mann et al. 2011;Sargeant et al. 2011;Eva et al. 2012;Watling et al. 2013aWatling et al. , 2014Watling 2014aWatling , 2014bWatling , 2016Ramani, et al. 2017aRamani, et al. , 2017bRamani, et al. , 2019. Self-assessment and reflection are also central to feedback seeking and acceptance (Sargeant et al. 2008b(Sargeant et al. , 2009K€ onings et al. 2016). In medical education, use of vague feedback language is common as faculty wish to avoid threats to friendly working relationships, learner self-esteem and autonomy (Sargeant et al. 2007(Sargeant et al. , 2008avan de Ridder et al. 2015avan de Ridder et al. , 2015bRamani et al. 2018). ...
... Teachers need to support as well as challenge learners to help them grow (Daloz and Daloz 1999). Since unguided self-assessment is often inaccurate (Regehr and Eva 2006;Regehr 2008, 2011) and learners reject feedback that conflicts with their self-assessment (Mann et al. 2011), teachers can guide accurate calibration of learner performance through facilitated reflection and informed self-assessment using data from multiple sources (Boud 1995;Sargeant et al. 2008bSargeant et al. , 2010. In summary, teachers can enhance learners' self-efficacy by discussing previous successes, reinforcing behaviours that contributed to success (support), encouraging them to tackle new and complex tasks (challenge), and providing constructive feedback to assist them in succeeding at new tasks. ...
... Awareness of one's abilities, assumptions and reactions can help learners understand how their actions are perceived by others including patients and handle challenging clinical situations better (Novack et al. 1997(Novack et al. , 1999. Strategies that teachers can use to develop learner self-awareness include: application of the Johari window model to feedback conversations (Luft 1969;Ramani et al. 2017a), developing an educational alliance, facilitating reflection of performance, debriefing challenging clinical situations, sharing own challenges and limitations and stimulating self-discovery (Sargeant et al. 2008a(Sargeant et al. , 2008b(Sargeant et al. , 2010Sargeant 2015;Telio et al. 2015Telio et al. , 2016Armson et al. 2019). ...
Article
This AMEE guide provides a framework and practical strategies for teachers, learners and institutions to promote meaningful feedback conversations that emphasise performance improvement and professional growth. Recommended strategies are based on recent feedback research and literature, which emphasise the sociocultural nature of these complex interactions. We use key concepts from three theories as the underpinnings of the recommended strategies: sociocultural, politeness and self-determination theories. We view the content and impact of feedback conversations through the perspective of learners, teachers and institutions, always focussing on learner growth. The guide emphasises the role of teachers in forming educational alliances with their learners, setting a safe learning climate, fostering self-awareness about their performance, engaging with learners in informed self-assessment and reflection, and co-creating the learning environment and learning opportunities with their learners. We highlight the role of institutions in enhancing the feedback culture by encouraging a growth mind-set and a learning goal-orientation. Practical advice is provided on techniques and strategies that can be used and applied by learners, teachers and institutions to effectively foster all these elements. Finally, we highlight throughout the critical importance of congruence between the three levels of culture: unwritten values, espoused values and day to day behaviours.
... Le connotazioni negative assunte in ambito nazionale dal termine "portfolio" risalgono all'aspro dibattito sulla utilità di questo strumento generato dalla Riforma Moratti 1 e la confusione in merito a utilizzo, conte-nuti e finalità fecero sì che il Garante della privacy ne decretasse la sua messa al bando (Ellerani, 2013). Tuttavia, alla luce dell'ampia letteratura internazionale sul valore della documentazione del proprio processo formativo -anche in contesti che esulano dall'istruzione, per esempio nell'ambito della formazione medica (Sargeant et al., 2008) -insieme alla riflessione che può scaturire dalla considerazione degli obiettivi più o meno raggiunti, si impone una seria riconsiderazione di questo strumento (Snadden & Thomas, 1998;Nicol & Milligan, 2006;Birgin & Baki, 2007;Thomas, Martin & Pleasants, 2011;Murphey, 2017 per citare solo alcuni studi). Anche in ambito nazionale non mancano pubblicazioni che continuano a promuovere l'uso del portfolio a tutti i livelli di scolarizzazione, come per esempio quelle di Mariani (2004), Pedrizzi (2007) e Cisotto (2011. ...
... Per accrescere la qualità della autovalutazione, tuttavia, è necessario offrire agli studenti più fonti di dati, credibili e specifici, che facilitino la loro riflessione. Sargeant et al. (2008), che hanno studiato l'autovalutazione in campo medico, suggeriscono, appunto, il ricorso a un multisource feedback (che può essere ottenuto facendo ricorso a check list, rubriche, peer feedback) in corrispondenza dei molteplici fattori che incidono sulla capacità autovalutativa: ...
... By providing external stimuli that compare different quality of work within a common topic or element, students were more likely to notice differences, pick up on features and key information that they might otherwise overlook, and gain deeper understanding of a subject. In the following section, we discuss why providing contrasting cases that highlight the most important elements that differentiate a poor from a good answer would enhance self-assessment (Andrade, Du, & Mycek, 2010;Galbraith, Hawkins, & Holmboe, 2008;Sargeant, Mann, VanDer Vleuten, & Metsemakers, 2008;Schraw, 2009;Schwartz, Chase, Oppezzo, & Chin, 2011;Schwartz & Bransford, 1998;Schwartz & Martin, 2004). ...
... Since current (electrons moving) is not observable with the human eye, students must rely on experimental measurements to help develop a mental model of how current divides across the branches while the total remains the same. As evidenced in past studies where the self-assessment supports were utilized, being able to accurately reflect upon key information and supporting evidence can help students develop a more accurate mental model of abstract concepts in science (Azevedo, 2005b;Lin, Hmelo, Kinzer, & Secules, 1999;Nokes, Hausmann, VanLehn, & Gershman, 2010;Sargeant et al., 2008;Schwartz et al., 2011;Warren, 2010). Thus, we hypothesized providing contrasting case supports would be effective in increasing students' performance and self-assessment accuracy by making difficult to visualize concepts, like current, more recognizable. ...
Full-text available
Article
Self-assessment is essential to scientific literacy as stated by the National Research Council Committee on Conceptual Framework for the New K-12 Science Education Standards and has since been incorporated into the Next Generation Science Standards. However, little empirical evidence documents which instructional tools are beneficial in improving students’ self-assessment in science learning. As such, we conducted a classroom-based quasi- experiment to test whether contrasting case-based instructional supports improved 390 introductory high school physics students’ accuracy in self-assessment and academic performance when solving physics problems. We compared the effects of providing students with content knowledge and contrasting cases (contrast- ing good and poor solution) on students’ physics problem solving and their ability to self-assess, with the effects of either present- ing them with content knowledge and good solutions or teaching only content knowledge. We found contrasting examples improved the accuracy of students’ self-assessments, promoted content learning, and enhanced the development of self-assessment strategies.
... Le connotazioni negative assunte in ambito nazionale dal termine "portfolio" risalgono all'aspro dibattito sulla utilità di questo strumento generato dalla Riforma Moratti 1 e la confusione in merito a utilizzo, contenuti e finalità fecero sì che il Garante della privacy ne decretasse la sua messa al bando (Ellerani, 2013). Tuttavia, alla luce dell'ampia letteratura internazionale sul valore della documentazione del proprio processo formativo -anche in contesti che esulano dall'istruzione, per esempio nell'ambito della formazione medica (Sargeant et al., 2008) -insieme alla riflessione che può scaturire dalla considerazione degli obiettivi più o meno raggiunti, si impone una seria riconsiderazione di questo strumento (Snadden & Thomas, 1998;Nicol & Milligan, 2006;Birgin & Baki, 2007;Thomas, Martin & Pleasants, 2011;Murphey, 2017 per citare solo alcuni studi). Anche in ambito nazionale non mancano pubblicazioni che continuano a promuovere l'uso del portfolio a tutti i livelli di scolarizzazione, come per esempio quelle di Mariani (2004), Pedrizzi (2007) e Cisotto (2011). ...
... Per accrescere la qualità della autovalutazione, tuttavia, è necessario offrire agli studenti più fonti di dati, credibili e specifici, che facilitino la loro riflessione. Sargeant et al. (2008), che hanno studiato l'autovalutazione in campo medico, suggeriscono, appunto, il ricorso a un multisource feedback (che può essere ottenuto facendo ricorso a check list, rubriche, peer feedback) in corrispondenza dei molteplici fattori che incidono sulla capacità autovalutativa: ...
Full-text available
Article
ENGLISH This essay illustrates the teaching experience on a specialist English laboratory course for first and second year students on a Degree course in Pedagogical Sciences at Verona University (2016–2017 academic year) and presents a digital tool (Padlet) that students used to set up their portfolio. The positive impacts on learning generated by self- assessment through the use of this tool encourage a wider use of narrative tools (Zanchin, 2013) to promote a perspective of one’s own learning outcomes (Pellerey, 2013). The essay briefly presents the state of the art in the field of self—assessment. It then illustrates the instrument used during the experience and the data collected. The self—assessing, periodic, but systematic process promoted awareness of learning outcomes well before the final exam, thus leading to targeted recovery strategies and maintaining motivation through the whole course (Dörnyei, 2007). On the basis of the above, it can be argued that Padlet (or a similar online resource) may be successfully used to promote self—assessment in any educational and training context. ITALIAN Il saggio illustra l’esperienza di insegnamento nel laboratorio di inglese specializzato per studenti del primo e secondo anno della laurea magistrale in Scienze pedagogiche presso l’Università degli Studi di Verona nell’a.a. 2016—17 e presenta uno strumento digitale cui gli studenti hanno fatto ricorso per predisporre il proprio portfolio. Le ricadute positive sull’apprendimento generate dal ricorso alla autovalutazione attraverso l’utilizzo di questo strumento incoraggiano un più diffuso avvalersi di strumenti narrativi (Zanchin, 2013) per promuovere una visione prospettica del proprio apprendimento (Pellerey, 2013). Il saggio presenta brevemente lo stato dell’arte nel campo della autovalutazione. Quindi, illustra lo strumento utilizzato durante l’esperienza (Padlet) e i dati raccolti. Il processo autovalutativo, periodico ma sistematico, ha promosso consapevolezza degli esiti del proprio apprendimento ben prima dell’esame finale, inducendo in tal modo eventuali strategie di recupero mirate e mantenendo vivace la motivazione all’impegno (Dörnyei, 2007). Per tutti questi motivi si suggerisce l’uso di Padlet (o di una risorsa online analoga) per la promozione della autovalutazione in qualsiasi contesto educativo e formativo.
... 4 Physicians can identify these by assessing (and reflecting on) their own performance, yet research proves accurate and reliable self-assessment to be difficult. 7,8 Therefore, external feedback is a widely accepted and validated method to review performance and identify improvement opportunities. 7,9,10 External feedback is most likely accepted and applied to practice when it originates from a credible source. ...
... 7,8 Therefore, external feedback is a widely accepted and validated method to review performance and identify improvement opportunities. 7,9,10 External feedback is most likely accepted and applied to practice when it originates from a credible source. Research has found that physicians consider their peers as credible and valuable sources for feedback. ...
Article
Introduction: For continuous professional development, it is imperative that physicians regularly receive performance feedback from their peers. Research shows that professionals are more proactive in learning and knowledge sharing with peers in teams with more psychological safety. Psychological safety has however not been studied in relation to peers' performance feedback. This study investigated the association between physicians' perceptions of psychological safety and performance feedback received from their peers. Methods: We invited physicians of cardiology, gastroenterology, obstetrics and gynecology, otorhinolaryngology, pulmonology, neurology, and neurosurgery departments of an academic medical center to participate. Physicians evaluated psychological safety using Edmondson's seven-item validated scale and performance feedback using the adapted four-item feedback subscale of the validated System for Evaluation of Teaching Qualities, including corrective and positive feedback, explanations of feedback, and suggestions for improvement from peers. We analyzed the data using multilevel linear regression analyses adjusted for physicians' sex, years since being certified a medical specialist, and months working in the clinic under the study. Results: This study included 105 physicians (86.8% participated). Psychological safety was positively associated with physicians' perceptions of performance feedback from peers (B = 0.54, 95% confidence interval = 0.34-0.73, P-value <.001). Conclusions: Physicians who experienced more psychological safety were more likely to receive corrective and positive performance feedback from peers, explanations of feedback, and suggestions for improvement. Medical teams should consider investing in psychological safety to encourage performance feedback from peers, and thus support physicians' continuous professional development and their efforts to provide high-quality patient care.
... The modified MSLQ can serve as a measure of medical students' reflection on their learning, since it can provide teachers with indications of whether the students have appropriate motivation to initiate reflection and whether they have enough confidence, since the level of confidence influences their reflection on their learning [10,32,33]. It can also be used to examine whether they use the metacognitive skills to regulate and reflect on their learning and whether they seek and incorporate external feedback to inform their reflection [10,32,34,35]. ...
... The last component is feedback-seeking behaviour. Reflection cannot be an individual's isolated activity, since the results of self-assessing process tend to be inaccurate [8,32,35,[44][45][46]. Reflection process involves the process of processing and incorporating external data, one of which was in the form of feedback, to inform the reflection [6,9,34,47,48]. ...
Full-text available
Article
Background Reflection on learning is an essential component of effective learning. Deconstructing the components of reflection on learning using a self-regulated learning (SRL) framework, allows the assessment of students’ ability to reflect on their learning. The aim of this study was to validate an instrument to measure medical students’ reflection on their learning. Methods A systematic search was conducted to identify the most suitable instrument to measure students’ reflection on their learning based on the theoretical framework of SRL. The search identified the Motivated Strategies for Learning Questionnaire (MSLQ) which contained five subscales: internal goal orientation, self-efficacy, critical thinking, metacognitive/self-regulation, help seeking and peer learning. Using the original MSLQ as the foundation, we carried out three phases of a research program to develop a useful set of items: an expert panel’s review of items, a substantial pilot study, and a factor analysis of ratings of a modified set of items by preclinical and final year medical students. Results The factor analysis of the Modified MSLQ extracted four subscales with reasonable internal consistency: self-orientation, critical thinking, self-regulation and feedback-seeking. Each subscale correlates highly with the Modified MSLQ score, with modest inter-correlations between the subscales suggesting that they are measuring different components of the total score. Conclusion Medical students and their educators need to be able to monitor their learning in their complex academic and clinical environments. The Modified MSLQ provides a means of investigating and tracking individual medical students’ reflections on their learning.
... In three studies, negative feedback was reported as a barrier for A C C E P T E D physicians to engage in the SDL program 1,24,27 Moreover, a number of studies identified the perceived accuracy and credibility of the feedback as facilitators for physicians to adopt the given feedback. 1,27,28,30 Barriers for physicians to engage in feedback were also coded in the emotions domain (see Supplemental Digital Appendix 4 at http://links.lww.com/ACADMED/A547). In particular, Eva et al 30 explored the factors influencing responsiveness to feedback. ...
... barriers that may foil the efforts of even the most well-intentioned and well-informed of educators to provide meaningful feedback." Sargeant et al,27 in a study looking at directed self-assessment and feedback, also suggested professional culture as a barrier to SDL. However, a deeper understanding of what this culture constitutes and how it acts as a barrier to the behavior of physicians in regard to engaging in SDL is still underdeveloped. ...
... In three studies, negative feedback was reported as a barrier for A C C E P T E D physicians to engage in the SDL program 1,24,27 Moreover, a number of studies identified the perceived accuracy and credibility of the feedback as facilitators for physicians to adopt the given feedback. 1,27,28,30 Barriers for physicians to engage in feedback were also coded in the emotions domain (see Supplemental Digital Appendix 4 at http://links.lww.com/ACADMED/A547). In particular, Eva et al 30 explored the factors influencing responsiveness to feedback. ...
... barriers that may foil the efforts of even the most well-intentioned and well-informed of educators to provide meaningful feedback." Sargeant et al,27 in a study looking at directed self-assessment and feedback, also suggested professional culture as a barrier to SDL. However, a deeper understanding of what this culture constitutes and how it acts as a barrier to the behavior of physicians in regard to engaging in SDL is still underdeveloped. ...
Full-text available
Article
Purpose: This scoping review explored the barriers and facilitators that influence engagement in and implementation of self-directed learning (SDL) in continuing professional development (CPD) for physicians in Canada. Method: This review followed the six-stage scoping review framework of Arksey and O'Malley and of Daudt et al. In 2015, the authors searched eight online databases for English-language Canadian articles published January 2005-December 2015. To chart and analyze data from the 17 included studies, they employed a two-step analysis process composed of conventional content analysis followed by directed coding applying the Theoretical Domains Framework (TDF). Results: Conventional content analysis generated five categories of barriers and facilitators: individual, program, technological, environmental, and workplace/organizational. Directed coding guided by the TDF allowed analysis of barriers and facilitators to behavior change according to two key groups: physicians engaging in SDL, and SDL developers designing and implementing SDL programs. Of the 318 total barriers and facilitators coded, 290 (91.2%) were coded for physicians and 28 (8.8%) for SDL developers. The majority (209; 65.7%) were coded in four key TDF domains: environmental context and resources, social influences, beliefs about consequences, and behavioral regulation. Conclusions: This scoping review identified five categories of barriers and facilitators in the literature and four key TDF domains where most factors related to behavior change of physicians and SDL developers regarding SDL programs in CPD were coded. There was a significant gap in the literature about factors that may contribute to SDL developers' capacity to design and implement SDL programs in CPD.
... Physicians' performance is most likely to improve when CPD activities are tailored to individual performance gaps and learning needs (Melnick, 2004). Physicians can identify these by assessing (and reflecting on) their own performance, yet research proves accurate and reliable self-assessment to be difficult (Davis et al., 2006;Sargeant et al., 2008). Therefore external feedback is a widely accepted and validated method to review performance and identify improvement opportunities (Sargeant et al., 2008;Sargeant et al., 2013;Silver et al., 2008). ...
... Physicians can identify these by assessing (and reflecting on) their own performance, yet research proves accurate and reliable self-assessment to be difficult (Davis et al., 2006;Sargeant et al., 2008). Therefore external feedback is a widely accepted and validated method to review performance and identify improvement opportunities (Sargeant et al., 2008;Sargeant et al., 2013;Silver et al., 2008). External feedback is most likely accepted and applied to practice when it originates from a credible source. ...
Full-text available
Thesis
This thesis aimed to add a more detailed understanding of physician performance form a physician point of view. Being interested in physicians' perceptions, feelings, behaviour, relations to and interactions with eacht other, we collected their narratives and reflections. In this thesis I conclude that the essence of physician performance is shaped by Calling (i.e. having a career that provides a sense of meaning or purpose and is used to help others) and Comardeship (i.e. positove and supporting relationships based on mutual trust, safety and responsibility for each other). Indiviudal physicians can only bloom when they feel committed and dedicated in an environment that breathes a collaborative mindset, where sharing is about caring and mutual trust, cohesion and peer-support are felt. This thesis also concludes that the increasing clerical burden is a serious threat to physicians' values and essential drivers for performance. A clerical burden that results from a more commercial trend in healthcare, putting the spotlight on command, rules, process, accountability and efficiency. Based on the findings in my thesis, I strongly advocate to counter the current commercialisation climate by putting people in the spotlight ahead of control, rules and registration. Humanized, personalized, value based healthcare can only be achieved by putting the values of those who deliver that care first.
... However, selfassessment is currently recognized as an important and integral part of residency education. [42][43][44][45] Self-assessment processes often incorporate feedback from peers and faculty 46 and, when effectively communicated, can teach residents how to view feedback as a necessary part of ongoing improvement. The resident-centered RAFT process brought together the recognized benefits of resident self-assessment with collaborative dialogue, and was designed with the following goals: i) Resist traditional hierarchical relationships among learners and faculty by enacting collaborative processes of educational planning and assessment. ...
Full-text available
Article
Because healthcare knowledge, practices and systems change so rapidly, physicians-in-training need to develop skills related to lifelong learning. The adult learning paradigm defines the effective professional learner as autonomous and activated. A part of the residency’s p4 (Preparing the Personal Physician for Practice) redesign, the RAFT (Resident Assessment Facilitation Team) process was designed to encourage residents’ adoption of activated learning behaviors by incorporating their participation in team-based educational planning and assessment of competence. This study examined interaction within the RAFT meetings to guage residents’ adoption of activated learning behaviors over time. In this study, transcripts of RAFT meetings from a single cohort of residents during the first and third years of training were analyzed using a grounded theory approach. Teams of at least two analysts per transcript engaged in two rounds of descriptive coding and three levels of axial coding to examine interaction during the RAFT meetings and identify how that interaction was related to residents’ activated learning. Four categories of interaction were identified: advising, managing the process of the meeting, expressing and managing emotion, and demonstrating reflective practice and mindfulness. Across those categories, 36 sub-types of messages indicated both similarities and differences between first- and third-year residents. Specifically, third-year residents took a more active leadership role in meetings; faculty team members were more likely to hand over responsibility for problem solving to thirdyear residents. Integrating residents into the RAFT advising and assessment process provides a context for residents to practice and exhibit activated learning behaviors.
... However, the kind of feedback solicited in this study differs in at least three ways: first, it is the manager who requests feedback from their subordinates (the nurses) in order to make adjustments to the procedures of bedside handover; second, feedback is provided in a group session rather than individually; and, third, though it is about performance, the feedback is not directed to any one person in particular. Despite these contextual differences, some parallels have been found between the kind of feedback presented in this study and the feedback described as, for example, self-assessment in Sargeant, Mann, van der Vleuten, and Metsemakers (2008), from which this study draws on where appropriate to make up for this gap in the literature and to provide as wide an understanding of how male and female nurses provide feedback in staff meetings as possible. ...
Chapter
There is a continual increase of research in the field of linguistic (im)politeness, but classroom discourse has been largely overlooked as a source of data. The present chapter addresses this gap in the research by providing an exploration of linguistic (im)politeness in classroom discourse, based on linguistic (im)politeness theory as its theoretical underpinning and Conversation Analysis as its analytical framework. It demonstrates how both male and female lower-secondary English as a foreign language teachers use stereotypically feminine negative politeness strategies in the form of pre-reproach questions to establish and maintain classroom order while simultaneously developing and protecting interpersonal relationships (in a ‘context of care’) with their pupils. This is of particular interest in the context of de-gendering professional workplaces because gendered beliefs do still appear to be an overriding variable which influences teachers’ classroom management practices. Findings raise awareness of underlying mechanisms of gender and (im)politeness in classrooms by showing how participants’ linguistic, multimodal and sequential resources function in the interaction. It is argued that the current debate on teacher gender should include not only primary but also secondary school teaching and aim at challenging gender stereotypes in order to attract more prospective teachers and guarantee best practice at all educational levels.
... 33 A mentor can help to deal with the emotional aspects of the multisource feedback and to structure individual reflection and follow-up. 34 Use of multisource feedback and mentoring systems could thus help countries transitioning from a system based on self-assessments to 'directed' self-assessments as suggested by Sargeant et al. 35 Second, most systems relied on self-assessments and lifelong learning activities doctors selected themselves without attending to external assessment of practice performance. More specifically, by relying on credit Open Access accumulation systems that allowed doctors to choose their learning activities, 36 it was entirely at the doctors' discretion to judge their performance and learning needs. ...
Full-text available
Article
Objectives With increased cross-border movement, ensuring safe and high-quality healthcare has gained primacy. The purpose of recertification is to ensure quality of care through periodically attesting doctors’ professional proficiency in their field. Professional migration and facilitated cross-border recognition of qualifications, however, make us question the fitness of national policies for safeguarding patient care and the international accountability of doctors. Design and setting We performed document analyses and conducted 19 semistructured interviews to identify and describe key characteristics and effective components of 10 different European recertification systems, each representing one case (collective case study). We subsequently compared these systems to explore similarities and differences in terms of assessment criteria used to determine process quality. Results Great variety existed between countries in terms and assessment formats used, targeting cognition, competence and performance (Miller’s assessment pyramid). Recertification procedures and requirements also varied significantly, ranging from voluntary participation in professional development modules to the mandatory collection of multiple performance data in a competency-based portfolio. Knowledge assessment was fundamental to recertification in most countries. Another difference concerned the stakeholders involved in the recertification process: while some systems exclusively relied on doctors’ self-assessment, others involved multiple stakeholders but rarely included patients in assessment of doctors’ professional competence. Differences between systems partly reflected different goals and primary purposes of recertification. Conclusion Recertification systems differ substantially internationally with regard to the criteria they apply to assess doctors’ competence, their aims, requirements, assessment formats and patient involvement. In the light of professional mobility and associated demands for accountability, we recommend that competence assessment includes patients’ perspectives, and recertification practices be shared internationally to enhance transparency. This can help facilitate cross-border movement, while guaranteeing high-quality patient care.
... It has been reported that "self" rating objectivity is difficult because it is an emotional response. 8 Sargeant and colleagues 9 suggest that there are four phases of self-assessment: 1) receiving feedback from external sources, 2) reflecting on feedback to reconcile with self-assessment, 3) plan to use feedback, and 4) use feedback for improvement and learning. Often during simulation training feedback from instructors can be important, and researchers have suggested that debriefing is beneficial. ...
Full-text available
Article
Objectives Emergent Cricothyroidotomy (EC) is a rarely used yet life‐saving procedure that it is important for an Emergency physician to master throughout his training. We evaluated the difference in utilization of a Manikin or Porcine model among EM resident when teaching EC. We also evaluated the difference in the models using two different commonly utilized and taught techniques: “scalpel‐finger‐bougie” (SFB) technique and the Melker Technique (MT). Methods This was a prospective crossover design. Instructions about the procedure were provided, study participants were randomly assigned to one of two groups and performed cricothyrotomy on both manikin and porcine simulators using both the Melker and the SFB techniques. Each group was started with the opposite technique on the second simulator than they started with on the first simulator. After the procedures, study participants completed a questionnaire that used the same format for both groups. All survey questions required a 7‐point Likert scale response. Confidence, difficulty, reality of the anatomy, and landmarks were compared. Results 15 emergency medicine residents participated in the study. Compared to the porcine‐first group, the manikin‐first group rated the anatomy more realistic (question 5) after their first attempt 6.29 versus 5.87 than their second attempt (using the porcine model) 4.29 versus 5.62 (main effect for attempts (1st vs. 2nd) P=0.027; interaction P=0.074). Interestingly, the porcine model‐first group rated the landmarks (question 6) significantly easier to find in both attempts (6.87 versus 5.86 on the first attempt) and (6.50 versus 5.57 on the second attempt) (P=0.012). 12 participants (80%) chose SFB as their preferred technique to use in real‐life scenario. Conclusion The Manikin model tends to be more realistic in cricothyrotomy simulation than the porcine model. Influence of fresh pig skin overlying the porcine model may require further research. Landmark identification during the procedure was easier on both models when participants start with the porcine model. Participants prefer the SFB over MT if faced with a real‐life scenario. This article is protected by copyright. All rights reserved.
... Since being a good doctor appears central to the self-concept of physicians, any feedback that threatens this ideal is difficult to reconcile. 20 Recent research regarding implicit-bias-related feedback also found that providing physicians and nurses with feedback regarding their implicit biases could conflict with an idealized version of their professional identity. 21 Feedback regarding an individual's negative implicit biases can provoke defensiveness, [22][23][24] leading recipients to avoid feedback altogether. ...
Full-text available
Article
Purpose: Learners and practicing health professionals may dismiss emotionally charged feedback related to self, yet little research has examined how to address feedback that threatens an individual's identity. The implicit association test (IAT) provides feedback to individuals regarding their implicit biases. Anticipating feedback about implicit bias might be emotionally charged for mental health professionals, this study explored their experience of taking the IAT and receiving their results, to better understand the challenges of identity-threatening feedback. Method: The researchers sampled 32 psychiatry nurses, psychiatrists, and psychiatric residents at Western University in Ontario, Canada, after they completed the mental illness IAT and received their results. Using constructivist grounded theory, semistructured interviews were conducted from April to October 2017 regarding participants' experience of taking the IAT. Using constant comparative analysis, transcripts were iteratively coded and analyzed for results. Results: While most participants critiqued the IAT and questioned its credibility, many also described the experience of receiving feedback about their implicit biases as positive or neutral. Most justified their implicit biases while acknowledging the need to better manage them. Conclusions: These findings highlight a feedback paradox, calling into question assumptions regarding self-related feedback. Participants' reactions to the IAT suggest that potentially threatening self-related feedback may still be useful to participants who question its credibility. Further exploration of how the feedback conversation influences engagement with self-related feedback is needed.
... Studies (Sargeant et al., 2008;Eva and Regehr, 2011) have shown that medical students and junior doctors possess inadequate self-assessment capabilities and state that the determinants that impact informed self-assessment are complicated. In the particular setting of the self-assessment of practical skills (Evans, Leeson and Petrie, 2007), prevailing evidence proposes that healthcare trainees' self-assessments are unreliable when compared with standard assessment of work. ...
Article
Aim This study seeks to evaluate the utility of video-assisted self-assessment in the attainment of basic life support (BLS) skills in medical students. Methods On procuring approval from the Institutional Ethics committee, third-year medical students were enrolled in the study. A large group interactive session on BLS skills was conducted followed by a demonstration of BLS skills on a manikin by the instructor. The students’ performance was recorded using an Apple iPad. Self-assessment was done by the students twice using a standard checklist – before and after watching the video recordings of their performance. The disparity in the self-assessment scores before and after watching their video was analyzed, which was compared with that of the instructors’ feedback, using students ‘t’ test and Mann Whitney test. Feedback about this assessment methodology was obtained using a questionnaire. Results Forty-one 3rd year medical students participated. A significant difference (<0.05) in the students pre and post video assessment scores was observed. All students agreed that watching the video recording of their skill performance helped them identify the errors better. Conclusions In conclusion, video-assisted assessment is an effective and a feasible method to improve self-assessment accuracy regarding clinical skills.
... Because self-assessment is strongly biased [45] and selfdirected learning needs scaffolding [46], programmatic assessment promotes a form of directed self-assessment [47] through coaching or mentoring. Mentoring has shown to have many positive effects in education [48]. ...
Article
Objective: This paper addresses how communication skills can best be assessed. Since assessment and learning are strongly connected, the way communication skills are best learned is also described. Results: Communication skills are best learned in a longitudinal fashion with ample practice in an authentic setting. Confrontation of behavior initiates the learning process and should be supported by meaningful feedback through direct observation. When done appropriately a set of (learned) communication skills become integrated skilled communication, being versatilely used in purposeful goal-oriented clinical communication. The assessment of communication skills should follow a modern approach to assessment where the learning function of assessment is considered a priority. Individual assessments are feedback-oriented to promote further learning and development. The resulting rich information may be used to make progression decisions, usually in a group or committee decision. Conclusion: This modern programmatic approach to assessment fits the learning of skilled communication well. Practice implications: Implementation of a programmatic assessment approach to communication will entail a major innovation to education.
... I hesitated to include it because of the risk of sneaking in global assessments of one's overall ability, self-esteem, and self-concept ("I'm good enough, I'm smart enough, and doggone it, people like me, " Franken, 1992), which do not seem relevant to a discussion of feedback in the context of learning. Research on global self-assessment, or self-perception, is popular in the medical education literature, but even there, scholars have begun to question its usefulness in terms of influencing learning and professional growth (e.g., see Sargeant et al., 2008). Eva and Regehr (2008) seem to agree in the following passage, which states the case in a way that makes it worthy of a long quotation: Self-assessment is often (implicitly or otherwise) conceptualized as a personal, unguided reflection on performance for the purposes of generating an individually derived summary of one's own level of knowledge, skill, and understanding in a particular area. ...
Full-text available
Article
This article is a review of research on student self-assessment conducted largely between 2013 and 2018. The purpose of the review is to provide an updated overview of theory and research. The treatment of theory involves articulating a refined definition and operationalization of self-assessment. The review of 76 empirical studies offers a critical perspective on what has been investigated, including the relationship between self-assessment and achievement, consistency of self-assessment and others' assessments, student perceptions of self-assessment, and the association between self-assessment and self-regulated learning. An argument is made for less research on consistency and summative self-assessment, and more on the cognitive and affective mechanisms of formative self-assessment.
... [1][2][3] Parallel to this is the increasing appreciation for the constructivist nature of the feedback process. [4][5][6] For example, individual perceptions of the feedback, 7 its alignment with selfassessment, 8 and trainee opinions of feedback providers 7,9 are important filters through which information is viewed and used. 7,10-12 Perceived credibility of feedback providers is a major determinant of how recipients view and incorporate feedback. ...
Article
Background : Resident perspectives on feedback are key determinants of its acceptance and effectiveness, and provider credibility is a critical element in perspective formation. It is unclear what factors influence a resident's judgment of feedback credibility. Objective : We examined how residents perceive the credibility of feedback providers during a formative objective structured clinical examination (OSCE) in 2 ways: (1) ratings of faculty examiners compared with standardized patient (SP) examiners, and (2) ratings of faculty examiners based on alignment of expertise and station content. Methods : During a formative OSCE, internal medicine residents were randomized to receive immediate feedback from either faculty examiners or SP examiners on communication stations, and at least 1 specialty congruent and either 1 specialty incongruent or general internist faculty examiner for clinical stations. Residents rated perceived credibility of feedback providers on a 7-point scale. Results were analyzed with proportional odds models for ordinal credibility ratings. Results : A total of 192 of 203 residents (95%), 72 faculty, and 10 SPs participated. For communication stations, odds of high credibility ratings were significantly lower for SP than for faculty examiners (odds ratio [OR] = 0.28, P < .001). For clinical stations, credibility odds were lower for specialty incongruent faculty (OR = 0.19, P < .001) and female faculty (OR = 0.45, P < .001). Conclusions : Faculty examiners were perceived as being more credible than SP examiners, despite standardizing feedback delivery. Specialty incongruency with station content and female sex were associated with lower credibility ratings for faculty examiners.
... sufficient to guide performance improvement effectively. Sargeant et al. (2008) proposed that it is important to integrate self-assessment with external feedback to enable its use for practice improvement. Stalmeijer et al. (2010) demonstrated that self-assessment combined with student feedback might at least provide faculty with an incentive to commit to change and seek support when they observed discrepancies between their own ratings and those of their students. ...
Full-text available
Article
Background: Previous studies have demonstrated that student ratings of a teachers’ performance do not incentivize clinical teachers to reflect critically and generate plans to improve their teaching. Peer group reflection might offer a solution in mediating this change. Aim: To investigate: (a) to which extent clinical teachers perceive self-evaluation, student ratings and peer group reflection effective; and (b) whether additional peer group reflection fosters critical reflection and the translation of feedback into concrete plans of action. Method: We conducted a quasi-experiment, inviting two groups of 10 clinical teachers each (1) to complete a self-evaluation and (2) subsequently examine their student ratings. One group participated in (3) an additional peer group reflection meeting. All participants were finally requested to define plans for improvement and evaluate each activity’s effectiveness. Results: Participants perceived all three activities to be effective. Levels of reflection did not differ across the two groups. However, participation in peer group reflection did result in generating more concrete plans to change clinical teaching. Conclusions: Peer group reflection on student ratings shows promise as tool to assist clinical teachers in generating plans for improvement. Future research should focus on whether teaching indeed improves with the introduction of peer group reflection.
... Even positive feedback lacks credibility; imposters discount positive feedback that portrays them as more competent than they feel. While attention has focused on the effect when corrective feedback is ignored, 40 we rarely consider the impact when positive feedback is discarded. ...
Article
Purpose: Mistakes are ubiquitous in medicine; when confronted by error, physicians may experience anxiety, guilt, and self-doubt. Feedback may be useful for navigating these feelings, but only if it matches a physician's self-assessment; self-doubt and the imposter syndrome are examples of inaccurate self-assessments that may affect receptivity to feedback. The impact of real or imagined underperformance on seemingly competent physicians is poorly understood. This study aimed to develop a deeper understanding in order to identify strategies to support all physicians who struggle. Method: In 2015, 28 practicing physicians were interviewed about their experiences with underperformance at an academic institution in Canada. Early in the data collection process, participants spontaneously identified the imposter syndrome as a feature of their experiences; questions about the imposter syndrome were probed in subsequent interviews. Results: Many participants-even those at advanced career stages-questioned the validity of their achievements; progressive independence and career advancement were variably experienced as "rising to the level of your incompetence." Not all participants identified as imposters; the imposter syndrome occurred at the extreme end of a spectrum of self-doubt. Even positive feedback could not buffer participants' insecurities, which participants rarely shared with their colleagues. Conclusions: Self-doubt variably affects clinicians at all career stages. Frequent transitions may cause a resurgence of self-doubt that may affect feedback credibility. Medical educators must recognize that it is not just the underperforming or failing learners who struggle and require support, and medical culture must create space for physicians to share their struggles.
... In the implementation of programmatic assessment, fostering a feedback seeking behaviour with a learning goal orientation (focused on growth and improvement) rather than a performance goal orientation (focused on gaining a positive judgement and garnering recognition of their own ability or on avoiding negative feedback) Teunissen and Bok 2013) may promote learners' acceptance and affect their perceptions of assessment as a continuum of stakes, ultimately facilitating the implementation of this principle. It has also been suggested that reflection is the path to acceptance of feedback that is out of line with learners' self-assessment (Sargeant et al. 2008). ...
Article
Introduction Programmatic assessment is a longitudinal, developmental approach that fosters and harnesses the learning function of assessment. Yet the implementation, a critical step to translate theory into practice, can be challenging. As part of the Ottawa 2020 consensus statement on programmatic assessment, we sought to provide descriptions of the implementation of the 12 principles of programmatic assessment and to gain insight into enablers and barriers across different institutions and contexts. Methods After the 2020 Ottawa conference, we surveyed 15 Health Profession Education programmes from six different countries about the implementation of the 12 principles of programmatic assessment. Survey responses were analysed using a deductive thematic analysis. Results and Discussion A wide range of implementations were reported although the principles remained, for the most part, faithful to the original enunciation and rationale. Enablers included strong leadership support, ongoing faculty development, providing students with clear expectations about assessment, simultaneous curriculum renewal and organisational commitment to change. Most barriers were related to the need for a paradigm shift in the culture of assessment. Descriptions of implementations in relation to the theoretical principles, across multiple educational contexts, coupled with explanations of enablers and barriers, provided new insights and a clearer understanding of the strategic and operational considerations in the implementation of programmatic assessment. Future research is needed to further explore how contextual and cultural factors affect implementation.
... [3][4][5][6][7][8][9][10] There has been extensive research emphasizing the sociocultural aspects of feedback and its impact on learners' behavior. [11][12][13][14] One critical aspect of feedback is the way of delivery, it is important for faculty to provide feedback that reinforces behaviors and skills, and further promotes critical thinking and areas of improvement. 15,16 Another important aspect is the approachability of the preceptor/educator; as described by Reddy et al 17 and Telio et al, 12 residents tend to avoid interactions with faculty members who do not create conducive environments for the feedback process. ...
Article
Introduction: Imperative to medical training is the observation and provision of feedback. In this era of competency-based medical education, feedback is one of the core components of this new model. A better understanding of the medical faculty's attitudes and experiences when providing feedback is essential. Currently, there are limited qualitative studies that have explored attitudes and experiences of faculty members when giving corrective feedback to medical trainees. Methods: To allow an in-depth exploration of this phenomenon, a hermeneutics phenomenology approach was used, by conducting semistructured interviews with 10 faculty members representing six disciplines and used thematic analysis to create data-driven codes and identify key themes through an iterative consensus-building process. Results: Four themes were identified by the authors: (1) Elements of effective feedback, (2) Faculty members' perception of giving corrective feedback, (3) Challenges as it relates to the assessment culture of the institution, and (4) Providing effective corrective feedback as a mutual process focused on relationship building between learners and preceptors. Discussion: By exploring faculty members' perceptions of providing perceived corrective feedback, we identified actionable recommendations based on the study participants' experiences, expectations, and challenges which could be addressed involving future faculty development with the focus on modifying concepts of feedback and institutional changes that will promote an attitudinal and a cultural shift.
... First, the MeRS-37 is a self-administered questionnaire and is inherently confined to a single selfassessment [42], thus bias could arise due to the selfenhancement [43]. Thus, the MeRS scores should be triangulated with other sources of evaluation is recommended and this approach will hopefully pay off in the long run [44]. Second, this study only confined to three categories of validity which were content, response process, and internal structure, therefore, future studies could be carried out to investigate other categories of validity which are the relationships of MeRS-37 with other variables and it's consequences [30]. ...
Full-text available
Article
Purpose Most of the resilience scales were developed for the non-medical population, therefore the purpose of this study was developing and validating a resilience scale for medical professionals – namely Medical Professionals Resilience Scale (MeRS). Methods A questionnaire development and validation study was conducted. The resilience domains and items were identified and generated through a literature review. The content validation was carried out by content experts and the content validity index (CVI) was calculated. The face validation was performed by medical officers and the face validity index (FVI) was calculated. The final MeRS was administered to 167 medical officers, exploratory factor analysis (EFA) and reliability analysis were performed to assess MeRS’s factorial structure and internal consistency. Results Four domains with 89 items of medical professionals’ resilience were developed. Following that, the content and face validation was conducted, and a total of 41-items remained for construct validation. EFA extracted four factors, namely growth, control, involvement, and resourceful, with a total of 37 items. The items’ CVI and FVI values were more than 0.80. The final MeRS’s items had factor loading values ranged from 0.41 to 0.76, and the Cronbach’s alpha values of the resilience domains ranged from 0.72 to 0.89. Conclusions MeRS is a promising scale for measuring medical professionals’ resilience as it showed good psychometric properties. This study provided validity evidence in terms of content, response process, and internal structure that supported the validity of MeRS in the measurement of resilience domains among medical professionals.
... 48,49 Self-directed learning requires considerable external direction and scaffolding. 50 Coaching has been shown to increase residents' abilities to recognize and reflect on learning opportunities as well as seek additional activities to improve performance. 51 The creation of a psychologically safe, interpersonal space in which so-called imposter syndrome and other doubts Competency-Based Assessment can be addressed will help trainees and their faculty and coaches move from a fixed to a growth mindset and become more resilient within environments that expose their inadequate skills. ...
Article
Medical education programs are failing to meet the health needs of patients and communities. Misalignments exist on multiple levels, including content (what trainees learn), pedagogy (how trainees learn), and culture (why trainees learn). To address these challenges effectively, competency-based assessment (CBA) for psychiatric medical education must simultaneously produce life-long learners who can self-regulate their own growth and trustworthy processes that determine and accelerate readiness for independent practice. The key to effectively doing so is situating assessment within a carefully designed system with several, critical, interacting components: workplace-based assessment, ongoing faculty development, learning analytics, longitudinal coaching, and fit-for-purpose clinical competency committees.
... 9 Such feedback promotes self-reflection to reconcile feedback with existing self-assessments, helping incorporate feedback into future practice. 10 While formative assessments can be collected via checklists, research demonstrates narrative feedback related to professionalism and communication is particularly valuable for guiding learners. 11 Reviewing narrative feedback enables development of programs such as workshops to teach faculty how to provide feedback and continuously improve it. ...
Full-text available
Article
Background: High-quality and high-utility feedback allows for the development of improvement plans for trainees. The current manual assessment of the quality of this feedback is time consuming and subjective. We propose the use of machine learning to rapidly distinguish the quality of attending feedback on resident performance. Methods: Using a preexisting databank of 1925 manually reviewed feedback comments from 4 anesthesiology residency programs, we trained machine learning models to predict whether comments contained 6 predefined feedback traits (actionable, behavior focused, detailed, negative feedback, professionalism/communication, and specific) and predict the utility score of the comment on a scale of 1-5. Comments with ≥4 feedback traits were classified as high-quality and comments with ≥4 utility scores were classified as high-utility; otherwise comments were considered low-quality or low-utility, respectively. We used RapidMiner Studio (RapidMiner, Inc, Boston, MA), a data science platform, to train, validate, and score performance of models. Results: Models for predicting the presence of feedback traits had accuracies of 74.4%-82.2%. Predictions on utility category were 82.1% accurate, with 89.2% sensitivity, and 89.8% class precision for low-utility predictions. Predictions on quality category were 78.5% accurate, with 86.1% sensitivity, and 85.0% class precision for low-quality predictions. Fifteen to 20 hours were spent by a research assistant with no prior experience in machine learning to become familiar with software, create models, and review performance on predictions made. The program read data, applied models, and generated predictions within minutes. In contrast, a recent manual feedback scoring effort by an author took 15 hours to manually collate and score 200 comments during the course of 2 weeks. Conclusions: Harnessing the potential of machine learning allows for rapid assessment of attending feedback on resident performance. Using predictive models to rapidly screen for low-quality and low-utility feedback can aid programs in improving feedback provision, both globally and by individual faculty.
... First, the MeRS-37 is a self-administered questionnaire and is inherently confined to a single selfassessment [42], thus bias could arise due to the selfenhancement [43]. Thus, the MeRS scores should be triangulated with other sources of evaluation is recommended and this approach will hopefully pay off in the long run [44]. Second, this study only confined to three categories of validity which were content, response process, and internal structure, therefore, future studies could be carried out to investigate other categories of validity which are the relationships of MeRS-37 with other variables and it's consequences [30]. ...
Full-text available
Preprint
Purpose: Most of the resilience scales were developed for the non-medical population, therefore the purpose of this study was developing and validating a resilience scale for medical professionals – namely Medical Professionals Resilience Scale (MeRS). Methods: A questionnaire development and validation study was conducted. The resilience domains and items were identified and generated through a literature review. The content validation was carried out by content experts and the content validity index (CVI) was calculated. The face validation was performed by medical officers and the face validity index (FVI) was calculated. The final MeRS was administered to 167 medical officers, exploratory factor analysis (EFA) and reliability analysis were performed to assess MeRS’s factorial structure and internal consistency. Results: Four domains with 89 items of medical professionals’ resilience were developed. Following that, the content and face validation was conducted, and a total of 41-items remained for construct validation. EFA extracted four factors, namely growth, control, involvement, and resourceful, with a total of 37 items. The items’ CVI and FVI values were more than 0.80. The final MeRS’s items had factor loading values ranged from 0.41 to 0.76, and the Cronbach’s alpha values of the resilience domains ranged from 0.72 to 0.89. Conclusion: MeRS is a promising scale for measuring medical professionals’ resilience as it showed good psychometric properties. This study provided validity evidence in terms of content, response process, and internal structure that supported the validity of MeRS in the measurement of resilience domains among medical professionals.
... 1,2 A variety of studies within and outside the health professions have suggested individuals are unable to accurately judge their abilities, 3 placing heavy emphasis on the availability of external data to facilitate growth in knowledge and skills. 4 Efforts to provide such data, however, must contend with at least three contextual realities: (1) busy health professionals are time constrained; ...
Article
Introduction: Using assessment to facilitate learning is a well-established priority in education but has been associated with variable effectiveness for continuing professional development. What factors modulate the impact of testing in practitioners are unclear. We aimed to improve capacity to support maintenance of competence by exploring variables that influence the value of web-based pretesting. Methods: Family physicians belonging to a practice-based learning program studied two educational modules independently or in small groups. Before learning sessions they completed a needs assessment and were assigned to either sit a pretest intervention or read a relevant review article. After the learning session, they completed an outcome test, indicated plans to change practice, and subsequently documented changes made. Results: One hundred twelve physicians completed the study, 92 in small groups. The average lag between tests was 6.3 weeks. Relative to those given a review article, physicians given a pretest intervention: (1) reported spending less time completing the assigned task (16.7 versus 25.7 minutes); (2) performed better on outcome test questions that were repeated from the pretest (65.9% versus 58.7%); and (3) when the learning module was completed independently, reported making a greater proportion of practice changes to which they committed (80.0% versus 45.0%). Knowledge gain was unrelated to physicians' stated needs. Discussion: Low-stakes formative quizzes, delivered with feedback, can influence the amount of material practicing physicians remember from an educational intervention independent of perceptions regarding the need to engage in continuing professional development on the particular topic.
... Documenting and making judgements about their own competence during training can provide an opportunity for residents to build skills in selfregulated learning-particularly when embedded in a program that incorporates guided self-assessment. In guided or directed self-assessment, 10 residents are taught to use external evidence to calibrate selfassessment of their own strengths and knowledge gaps. Instruction in these skills is delivered through both explicit means, such as comparing their own performance against assessment rubrics, 11 and implicit means, such as role-modeling by clinical teachers. ...
Article
Background and objectives: In competency-based medical education (CBME), should resident self-assessments be included in the array of evidence upon which summative progress decisions are made? We examined the congruence between self-assessments and preceptor assessments of residents using assessment data collected in a 2-year Canadian family medicine residency program that uses programmatic assessment as part of their approach to CBME. Methods: This was a retrospective observational cohort study using a learning analytics approach. The data source was archived formative workplace-based assessment forms (fieldnotes) stored in an online portfolio by family medicine residents and preceptors. Data came from three academic teaching sites over 3 academic years (2015-2016, 2016-2017, 2017-2018), and were analyzed in aggregate using nonparametric tests to evaluate differences in progress levels selected both within and between groups. Results: In aggregate, first-year residents' self-reported progress was consistent with that indicated by preceptors. Progress level rating on fieldnotes improved over training in both groups. Second-year residents tended to assign themselves higher ratings on self-entered assessments compared with those assigned by preceptors; however, the effect sizes associated with these findings were small. Conclusions: Although we found differences in the progress level selected between preceptor-entered and resident-entered fieldnotes, small effect sizes suggest these differences may have little practical significance. Reasonable consistency between resident self-assessments and preceptor assessments suggests that benefits of guided self-assessment (eg, support of self-regulated learning, program efficacy monitoring) remain appealing despite potential risks.
... A few researchers have additionally observed that self-idea and social assessment in the intellectual conduct model of social nervousness are the reasons for luxury brand behaviour. This model brings up that when people give a lot of consideration to the assessment of others, they will have uneasiness (Sargeant, Mann, van der Vleuten, & Metsemakers, 2008). Weeks and Howell (2012) likewise found that people with significant degrees of adverse appraisals of dread experienced more elevated levels of social anxiety. ...
Full-text available
Conference Paper
People take the opportunity from social-networking sites like Facebook, to express themselves in various communities. Various studies address that it has influence over self-esteem of its users. In this paper, we analyse how the self-esteem of a person is affected in the computational world. To accomplish this, we built a computational model of esteem. A questionnaire-based survey was conducted to collect data for model verification. Different simulation experiments were conducted to compare and evaluate the model concerning findings from the literature and data. This model can be used as a useful input to provide support to people who are influenced by negative feedback.
Article
Purpose: Feedback continues to present a challenge for competency-based medical education (CBME). Clear, consistent, and credible feedback is vital to supporting one's ongoing development, yet it can be difficult to gather clinical performance data about residents. This study sought to determine whether providing residents with electronic health record (EHR) based report cards, as well as an opportunity to discuss these data with faculty trained using the R2C2 model, can help residents understand and interpret their clinical performance metrics. Method: Using action research methodology, the author team collected EHR data from July 2017 to February 2020, for all residents (n = 21) in 1 5-year Emergency Medicine program and created personalized report cards for each resident. During October 6-17, 2020, 8 out of 17 eligible residents agreed to have their feedback conversations recorded and participate in a subsequent interview with a non-physician member of the research team. Data were analyzed using thematic analysis and the authors used inductive analysis to identify themes in the data. Results: In analyzing both the feedback conversations as well as the individual interviews with faculty and residents, the authors identified 2 main themes: (1) Reactions and responses to receiving personalized EHR data and (2) The value of EHR data for assessment and feedback purposes. All participants believed that EHR data metrics are useful for prompting self-reflection and many pointed to their utility in providing suggestions for actionable changes in their clinical practice. For faculty, having a tool through which underperforming residents can be shown "objective" data about their clinical performance helps underscore the need for improvement, particularly when residents are resistant. Conclusions: The EHR is a valuable source of educational data and this study demonstrates one of the many thoughtful ways it can be used for assessment and feedback purposes.
Article
Objective: Our previous home-video basic surgical skills curriculum required substantial faculty time and resources, and was limited by delayed feedback and technical difficulties. Consequently, we integrated that curriculum with a mobile application platform. Our purpose is to describe this application and learner satisfaction. Materials and methods: The mobile platform incorporates a patented pedagogical design based on Ericsson's deliberate practice and Bandura's social learning theory. Instructors built step-wise skills modules. During the challenge phase, learners watched a video of surgical tasks completed by experts and uploaded a video of themselves performing the same task. In the Peer Review phase, they used a grading rubric to provide feedback. In the Recap stage, learners received individual feedback and could review their own videos. Two groups of learners, graduating medical students and matriculating surgical residents, participated in this independent learning platform, along with 2 to 4 laboratory sessions, and completed a survey about their experience. Survey responses were summarized descriptively and comments analyzed using content analysis. Results: Fifty learners submitted videos of assigned tasks and completed peer reviews. Learners reported positive experiences specifically for the Peer Review Stage, structured home practice, ease of mobile access to submit and review videos, and ongoing immediate feedback. Over half of the learners reported spending at least 10 to 30 minute practicing skills before recording their videos and over 80% rerecorded at least 2 times before submission. Content analysis revealed learners engaged with the educational concepts designed into the platform. Conclusion: Learners easily used and were satisfied with a mobile-technology teaching platform that maintained the fundamental content, educational theories, and organizational structure of our previously effective surgical skills curriculum. Prior challenges were directly addressed through the mobile application's ease of use, support of deliberate practice, and improved timeliness of feedback.
Full-text available
Book
Covering both higher education and school education, this book contributes to the field of assessment by providing a systematic account of student self-assessment based on a consistent conceptualisation. Yan advocates viewing self-assessment as an active and reflective process and using it as a learning strategy rather than an assessment method. He builds on a newly developed self-assessment model adopting a process perspective and synthesises a series of interrelated empirical investigations into the whole “chain” of student self-assessment research. The research encompassed in the volume spans from self-assessment practices and measurement, through predictors of self-assessment, its interweaved relationship with self-regulated learning and feedback literacy, impact on student learning outcomes, to designing sustainable self-assessment interventions. The empirical evidence is from a wide range of current scholarship to ensure that the principles and implications conveyed are applicable internationally. Policymakers, students and scholars in educational assessment, educational psychology, and teaching and instruction will find the theoretical explorations and empirical investigations contained within useful, to show how student self-assessment could be better conceptualised, researched and practised.
Chapter
Competency-based assessment (CBA) for psychiatric graduate medical education must simultaneously optimize three goals: (1) Maximize learning through formative assessment (assessment for learning). (2) Enable robust and trustworthy high stakes decision-making (e.g., promotion or selection) through summative assessment (assessment of learning). (3) Support ongoing improvements in the curriculum. As such, the CBA system must promote both self-regulated growth and clinical competence as judged by a trustworthy process. The key to effectively doing so is situating these activities within a carefully designed system of assessment with several critical components: workplace-based assessment, ongoing faculty development, learning analytics, longitudinal coaching, and fit-for-purpose clinical competency committees.
Article
Background In 2013 the Accreditation Council for Graduate Medical Education (ACGME) introduced "Milestones" designed to nationally standardize the assessment of resident physicians. Previous studies compare resident self-assessment on milestones to faculty assessment, with varying degrees of agreement, but integration of self-assessment into the formative feedback process has not yet been directly studied. This study uses a conceptual framework of self-determination theory, integrated with concepts from adult learning theory, to compare the perception of the feedback quality given in semiannual reviews before and after the incorporation of resident self-assessment into the feedback process. Methods This was an interventional study conducted in a single Emergency Medicine residency program at a major academic hospital over one calendar year. Residents first engaged in a semiannual review without self-assessment. At subsequent semiannual reviews, residents completed a Milestone-based self-assessment which was provided to the faculty member assigned to conduct their semiannual review. Residents and faculty completed surveys rating perception of feedback quality. Two-sided Wilcoxon signed-rank tests were used in comparison analysis. Results One resident did not self-assess prior to the semiannual review and was excluded leaving 25 paired surveys for analysis. Residents found feedback after the self-assessment more actionable (p = .013), insightful (p = .010), and better overall (p = .025). Similarly, faculty felt the feedback they provided was more actionable (p < .001), more insightful (p < .001), better communicated (p < .001), led to improved resident understanding of milestones (p <.001), and were overall more satisfied (p < .001). Freetext comments explore pre- and post-intervention perceptions of feedback. Conclusions Integration of self-assessment into semiannual reviews improves perception of feedback given to residents as perceived by both residents and faculty. Although limited by sample size, the results are promising for a simple, evidence-based intervention to improve feedback during an existing mandated feedback opportunity.
Article
Phenomenon Learning is a complex phenomenon in which learners can vary in their learning orientation, learning approaches, and perception of the learning environment. Learners may be characterized as mastery oriented or performance oriented, and this learning orientation can influence their learning approaches, such as whether to seek feedback or ask for help. The learning environment includes institutional assessment policies and informal interactions that emphasize outcomes and differentiation among students. When learners perceive the learning environment as performance oriented, they feel competition from other learners and fear negative feedback. This complex interplay of learning orientation, its influence on feedback-seeking behavior, and the environmental influence and reinforcement of both can be captured through learner profiles. Approach In this multi-institutional cross-sectional survey study of students from six U.S. medical schools, we investigated learner profiles that characterize how these domains manifest together across individual learners. We then determined how these profiles are associated with students’ clerkship grades. Measures included the Patterns of Adaptive Learning for learning orientation, self-reports for learning approaches, and perceptions of the learning environment. We used latent profile analysis to cluster students who share common characteristics around orientation, behavior, and environment. The relationship between these profiles and the percentage of honors earned was examined using multiple regression analysis. Findings We found four distinct learner profiles within 666 of 974 (68%) analyzable responses: (1) performance oriented with negative perceptions of environment, (2) mastery oriented with desirable learning approaches, (3) “average” group, and (4) mastery orientation only. Profile 1 (M = 39%) and Profile 4 (M = 38%) students received fewer clerkship honors compared with other profiles. Profile 2 students earned a significantly higher percentage of honors grades (M = 54%) compared with other students. Profile 3 students (average group) earned the second highest percentage of honors (M = 46%). Effect sizes comparing the percentage of honors varied from 0.32 to 0.65, indicating medium to large differences. Insights The highest performers, as measured by clerkship honors grades, were mastery oriented, perceived a more positive learning environment, and were comfortable asking questions and seeking feedback for their learning. The lowest performing students were performance oriented and had negative perceptions of their learning environments. Contrary to previous findings, we found that the relationship between mastery orientation and learning approaches is not unidirectional but rather seems to be mediated by perceptions of the learning environment.
Article
Introduction: Little is known of processes by which feedback affects learners to influence achievement. This review maps what is known of how learners interact with feedback, to better understand how feedback affects learning strategies, and to explore enhancing and inhibiting factors. Methods: Pilot searching indicated a wide range of interpretations of feedback and study designs, prompting the use of scoping methodology. Inclusion criteria comprised: (i) learners (undergraduate, postgraduate, continuing education) who regularly receive feedback, and (ii) studies that associated feedback with subsequent learner reaction. The screening was performed independently in duplicate. Data extraction and synthesis occurred via an iterative consensus approach. Self-regulatory learning theory (SRL) was used as the conceptual framework. Results: Of 4253 abstracts reviewed, 232 were included in the final synthesis. Understandings of feedback are diverse; a minority adopt recognised definitions. Distinct learner responses to feedback can be categorized as cognitive, behavioural, affective, and contextual with complex, overlapping interactions. Importantly emotional responses are commonplace; factors mediating them are pivotal in learner recipience. Conclusion: Feedback benefits learners most when focussed on learner needs, via engagement in bi-directional dialogue. Learner emotions must be supported, with the construction of positive learner-teacher relationships. A developmental agenda is key to learner's acceptance of feedback and enhancing future learning.
Chapter
Then, in this chapter the same steps are done for individuals who failed to succeed. Both perspectives address the book’s major concern that individual and social contributions to professional development cannot be thought separately.
Full-text available
Book
In our third volume of the Education Theory Made Practical series, we continue to try to close the gap betwen the theoretical world of health professions education and the real world of clinical and classroom teaching. Originally written as a joint venture between the International Clinician Educator blog (https://icenetblog.royalcollege.ca/) and the Academic Life in Emergency Medicine Faculty Incubator program (https://www.aliem.com/faculty-incubator/), this book is a great primer for those seeking to apply theory to their day-to-day teaching. The series was originally posted as a blog series and was subject to open peer review. This book is an edited compendium of these peer reviewed works, which have been reformatted and laid out in a textbook format.
Article
Objectives: Accurate self-assessment is an important aspect of practice-based learning and improvement and a critical skill for resident growth. The Accreditation Council for Graduate Medical Education mandates semiannual milestones assessments by a clinical competency committee (CCC) for all ophthalmology residents. There are six core competencies: patient care (PC), medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication skills. These competencies are assessed by the milestones rubric, which has detailed behavioral anchors and are also used for trainee self-assessments. This study compares resident self-assessed (SA) and faculty CCC milestones scores. Design: Residents completed milestones self-assessments prior to receiving individual score reports from the CCC. Correlation coefficients were calculated comparing the SA and CCC scores. In addition, statistical models were used to determine predictors of disparities and differences between the SA and CCC scores. Setting: Wilmer Eye Institute, Johns Hopkins Hospital. Participants: Twenty-one residents in the Wilmer Ophthalmology Residency program from July 2014 to June 2016. Results: Fifty-seven self-assessments were available for the analysis. For each resident's first assessment, SA and CCC scores were strongly correlated (r ≥ 0.6 and p < 0.05) for four milestones, and not correlated for the remaining 20 milestones. In multivariable models, the SA and CCC scores are less disparate for medical knowledge and systems-based practice competencies compared to practice-based learning and improvement. Higher year of training, PC and professionalism competencies were predictive of statistically significant resident overestimation of scores relative to the CCC. In addition, higher CCC scores predicted statistically significant lower SA-CCC disparities and differences. SA-CCC differences did not lower to a significant extent with repeated assessments or modification to the end-of-rotation evaluation forms. Conclusions: Self-assessments by ophthalmology residents are not well-correlated with faculty assessments, emphasizing the need for improved and frequent timely feedback. Residents have the greatest difficulty self-assessing their professionalism and PC competency. In general, senior residents and underperforming residents have more inaccurate self-assessments.
Article
Purpose Most medical students graduate without receiving feedback from a non-physician – a missed opportunity given the important role that interprofessional (IP) feedback plays in helping students develop IP competencies and increasingly IP clinical settings. To address this gap, we piloted a two-item, open-ended IP feedback survey for medical students. Methods We required 152 first-year medical students enrolled in a 15-month longitudinal clinical placement to request IP feedback twice. We used directed content analysis to examine feedback for quality (behavioral-orientation, specificity, valence, actionability) and content relevant to IP competencies. We then interviewed 14 students about their perceived usefulness of the feedback. Results The majority of feedback comments (85%) were behavior-oriented. Thirty-four percent of comments were constructive, of which 43% contained a specific strategy for improvement. Comments spanned the range of IP competency domains. Perceived usefulness was hindered by lack of student relationships with IP providers. Conclusions IP providers wrote specific, behavior-oriented feedback on collaboration skills that was comparable in quality to typical feedback in health professions education. However, limited student-provider relationships and perceived constructive feedback influenced student perceptions of its usefulness. To maximize usefulness of IP feedback, IP educational experiences should be designed to maximize frequency and depth of IP interactions, students should receive guidance on who to approach for IP feedback, and IP feedback should be reviewed with a faculty member.
Chapter
This SP Educator accidentally entered the field of Human Simulation in 1973 as a simulated patient, well before there was a sense of an emerging profession for SP Educators and before there was a community of practice for support and sharing. Over 40 years later, she reflects on her career, the changes she has seen in the field, and the implications of those changes for today’s SP Educators. Now, SP Educators are part of a maturing community of practice and have access to a large body of research and reference materials. Her narrative highlights the importance of SP Educators, regardless of their professional background, accessing this community of practice and these resources to create an intentional career.
Article
Once conceptualized as a static skill, self-assessment was believed to facilitate learning, foregrounding the role of accuracy. But when self-assessment is regarded as a dynamic process, less is known concerning how it unfolds and what differences there might be across students of different competency. To address this gap, the current study, situated in an undergraduate interpreting course, adopted a longitudinal and mixed-method design to record, capture and analyze the dynamics of the student voluntary self-assessment process as well as the sub-stages within. Analysis of variance was employed to process the quantitative data from a self-assessment practice scale, corroborated by qualitative analysis of student self-assessment reports and in-depth retrospective interviews. It was found that the students sought more external feedback through inquiry later in the semester. And highly competent students outperformed the least competent peers in this regard. But generally speaking, self-assessment practice of students was constant across time, and there was little significant difference among different competency groups. It attested to the potential of self-assessment implementation without formal guidance and regardless of student competency. But for self-assessment to be more rewarding, certain scaffolding is still necessary, particularly for students with intermediate competency.
Chapter
The purpose of assessment in mastery learning is to promote learner improvement. Assessment is a key and indispensable feature of the mastery learning bundle. Mastery learning approaches are linked to the ability to make accurate decisions. Accurate decisions depend on reliable data that have been collected using instruments and methods linked to curriculum learning objectives. This chapter has six sections that reveal the ways that instruction and assessment are inseparable in mastery learning: (a) curriculum and instruction, (b) validity argument, (c) assessment context, (d) assessment measures, (e) data, and (f) decisions. Health professions educators must collect validity evidence regularly to support the validity argument that they are making accurate decisions about learners at each step of assessment design and implementation.
Full-text available
Article
The use of multisource feedback as a management development tool is examined by integrating the empirical and theoretical literature on individual change from the fields of industrial/organizational psychology and clinical/counseling psychology. The assumptions underlying 360-degree feedback as a sufficient process of producing managerial change are questioned in terms of the theoretical and metaanalytic literature regarding the causes of personal change. It is argued that 360-degree feedback is best used as a springboard for management development. Lasting change is best achieved through an interdisciplinary coaching strategy involving what we know about adult development and change from industrial and clinical literature and practice. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Full-text available
Article
Executive Overview Performance feedback is an important part of many organizational interventions. Managers typically assume that providing employees with feedback about their performance makes it more likely that performance on the job will be improved. Despite the prevalence of feedback mechanisms in management interventions, however, feedback is not always as effective as is typically assumed. In this article, we present specific conditions under which feedback might be less effective, or even harmful. We then discuss the implications of our results and model for designing of interventions aimed at improving performance, and focus more narrowly on 360-degree appraisal systems. After arguing that these systems typically have design characteristics that reduce effectiveness, we conclude with recommendations for improving their effectiveness. We also emphasize the need for systematic evaluations of feedback interventions.
Full-text available
Article
This paper presents and organizes the results of two decades of research on feedback-seeking behavior according to three motives: the instrumental motive to achieve a goal, the ego-based motive to protect one’s ego, and the image-based motive to enhance and protect one’s image in an organization. Each motive is discussed with reference to its impact on the frequency of feedback seeking, seeking method (whether by inquiry or monitoring), timing of feedback seeking, choice of the target of feedback seeking, and the topic on which feedback is sought. The role of context in influencing these patterns is also discussed. Issues in the literature are identified throughout, and the review ends by identifying five promising areas for future research.
Full-text available
Article
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
Full-text available
Article
Effective clinical reasoning in nursing practice depends on the development of both cognitive and metacognitive skills. While a number of strategies have been implemented and tested to promote these skills, educators have not been able consistently to predict their development. Self-regulated learning theory suggests that this development requires concurrent attention to both the cognitive and metacognitive dimensions of reasoning in nursing care contexts. This paper reports on a study to explore the impact of self-regulated learning theory on reflective practice in nursing, and to advance the idea that both cognitive and metacognitive skills support the development of clinical reasoning skills. Integrative review of published literature in social science, educational psychology, nursing education, and professional education using the Cumulative Index to Nursing and Allied Health (CINAHL), Educational Resource Information Center (ERIC), and American Psychological Association (PsychInfo) Databases. The search included all English language articles with the key words clinical reasoning, cognition, critical thinking, metacognition, reflection, reflective practice, self-regulation and thinking. Reflective clinical reasoning in nursing practice depends on the development of both cognitive and metacognitive skill acquisition. This skill acquisition is best accomplished through teaching-learning attention to self-regulation learning theory. A critical analysis of the literature in the areas of critical thinking and reflective practice are described as a background for contemporary work with self-regulated learning theory. It is apparent that single-minded attention to critical thinking, without attention to the influence of metacognition or reflection, is but one perspective on clinical reasoning development. Likewise, single-minded attention to metacognition or reflection, without attention to the influence of critical thinking, is another perspective on clinical reasoning development. While strategies to facilitate critical thinking and reflective practice have been used in isolation from each other, there is evidence to suggest that they are inextricably linked and come together with the use of self-regulated learning prompts. Students and practising nurses are able to improve their cognitive and metacognitive skills in clinical contexts by using self-regulated learning strategies. The self-regulated learning model in nursing is offered to support teaching and learning of reflective clinical reasoning in nursing practice contexts.
Full-text available
Article
To identify existing instruments for rating peers (professional colleagues) in medical practice and to evaluate them in terms of how they have been developed, their validity and reliability, and their appropriateness for use in clinical settings, including primary care. Systematic literature review. Electronic search techniques, snowball sampling, and correspondence with specialists. The peer assessment instruments identified were evaluated in terms of how they were developed and to what extent, if relevant, their psychometric properties had been determined. A search of six electronic databases identified 4566 possible articles. After appraisal of the abstracts and in depth assessment of 42 articles, three rating scales fulfilled the inclusion criteria and were fully appraised. The three instruments did not meet established standards of instrument development, as no reference was made to a theoretical framework and the published psychometric data omitted essential work on construct and criterion validity. Rater training was absent, and guidance consisted of short written instructions. Two instruments were developed for a hospital setting in the United States and one for a primary care setting in Canada. The instruments developed to date for physicians to evaluate characteristics of colleagues need further assessment of validity before their widespread use is merited.
Full-text available
Article
A self-regulating profession holds its members accountable to the public it serves for the continuous development of the competencies they profess to hold. A central component of physician competence is professionalism, which requires lifelong learning that leads to improved performance in practice. A medical profession accomplishes accountability by providing its members periodic measurement of performance using reliable and valid instruments and judging performance against evidence-based standards, providing graduate and continuing medical education (CME) programs that advance members' knowledge and skills to meet these standards, and publicly certifying those who do so.
Full-text available
Article
Core physician activities of lifelong learning, continuing medical education credit, relicensure, specialty recertification, and clinical competence are linked to the abilities of physicians to assess their own learning needs and choose educational activities that meet these needs. To determine how accurately physicians self-assess compared with external observations of their competence. The electronic databases MEDLINE (1966-July 2006), EMBASE (1980-July 2006), CINAHL (1982-July 2006), PsycINFO (1967-July 2006), the Research and Development Resource Base in CME (1978-July 2006), and proprietary search engines were searched using terms related to self-directed learning, self-assessment, and self-reflection. Studies were included if they compared physicians' self-rated assessments with external observations, used quantifiable and replicable measures, included a study population of at least 50% practicing physicians, residents, or similar health professionals, and were conducted in the United Kingdom, Canada, United States, Australia, or New Zealand. Studies were excluded if they were comparisons of self-reports, studies of medical students, assessed physician beliefs about patient status, described the development of self-assessment measures, or were self-assessment programs of specialty societies. Studies conducted in the context of an educational or quality improvement intervention were included only if comparative data were obtained before the intervention. Study population, content area and self-assessment domain of the study, methods used to measure the self-assessment of study participants and those used to measure their competence or performance, existence and use of statistical tests, study outcomes, and explanatory comparative data were extracted. The search yielded 725 articles, of which 17 met all inclusion criteria. The studies included a wide range of domains, comparisons, measures, and methodological rigor. Of the 20 comparisons between self- and external assessment, 13 demonstrated little, no, or an inverse relationship and 7 demonstrated positive associations. A number of studies found the worst accuracy in self-assessment among physicians who were the least skilled and those who were the most confident. These results are consistent with those found in other professions. While suboptimal in quality, the preponderance of evidence suggests that physicians have a limited ability to accurately self-assess. The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment.
Article
Performance feedback is an important part of many organizational interventions. Managers typically assume that providing employees with feedback about their performance makes it more likely that performance on the job will be improved. Despite the prevalence of feedback mechanisms in management interventions, however, feedback is not always as effective as is typically assumed. In this article, we present specific conditions under which feedback might be less effective, or even harmful. We then discuss the implications of our results and model for designing of interventions aimed at improving performance, and focus more narrowly on 360-degree appraisal systems. After arguing that these systems typically have design characteristics that reduce effectiveness, we conclude with recommendations for improving their effectiveness. We also emphasize the need for systematic evaluations of feedback interventions.
Article
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of the participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
This study examined how 360° feedback ratings and self–other rating discrepancies related to reactions to feedback, perceptions of feedback accuracy, perceived usefulness of the feedback, and recipients' receptivity to development. The results indicated that less favorable ratings were related to beliefs that feedback was less accurate and to negative reactions. Negative reactions and perceptions that feedback was less accurate were related to beliefs that the feedback was less useful. Those who found feedback less useful were perceived by a facilitator as less development-focused. Goal orientation did not moderate the relationship between ratings and perceptions of accuracy or reactions to feedback. Goal orientation was related to perceptions of usefulness of the process several weeks after receipt of feedback. The results question widely held assumptions about 360° feedback that negative and discrepant feedback motivates positive change. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
Assessment within higher education has traditionally been the sole responsibility of the tutor. This paper considers the authors' experience with the introduction of both selfassessment and peer-assessment into a literature module at a new UK university. This paper describes the practical aspects and examines the outcomes. Reflections on the process are given for both students and tutor. It is concluded that, with careful application, the use of self-and peer assessment can yield positive educational benefits that more than justify any additional efforts required of students or tutor.
Article
The use of multisource feedback as a management development tool is examined by integrating the empirical and theoretical literature on individual change from the fields of industrial/organizational psychology and clinical/counseling psychology. The assumptions underlying 360-degree feedback as a sufficient process of producing managerial change are questioned in terms of the theoretical and metaanalytic literature regarding the causes of personal change. It is argued that 360-degree feedback is best used as a springboard for management development. Lasting change is best achieved through an interdisciplinary coaching strategy involving what we know about adult development and change from industrial and clinical literature and practice. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
We review evidence showing that multisource feedback ratings are related to other measures of leadership effectiveness and that different rater sources conceptualize performance in a similar manner. We then describe a meta-analysis of 24 longitudinal studies showing that improvement in direct report, peer, and supervisor ratings over time is generally small. We present a theoretical framework and review empirical evidence suggesting performance improvement should be more likely for some feedback recipients than others. Specifically, improvement is most likely to occur when feedback indicates that change is necessary, recipients have a positive feedback orientation, perceive a need to change their behavior, react positively to the feedback, believe change is feasible, set appropriate goals to regulate their behavior, and take actions that lead to skill and performance improvement.
Article
In industrialized societies, professionals have long been valued and set apart from other workers because of their specialized knowledge and skill. But has their role in these societies declined? Of what significance are they today? In this concise synthesis of the major debates about the professions since World War II, Eliot Freidson explores several broad questions about professionalism today—what it is, what its future is likely to be, and its value to public policy. Freidson argues that because professionalism is based on specialized knowledge, it is distinct from either bureaucratic or market-based forms of work. He predicts a rebirth of the professions during which practitioners lose some of their independence and become more accountable to standards of a professional elite. And, defending professionalism as a desirable method of providing complex, discretionary services to the public, Freidson argues that market-based or bureaucratic methods would impoverish the quality of service to consumers, and suggests ways the virtues of professionalism can be reinforced. The most accessible survey available of almost fifty years of theory and research by the scholar whose own work helped define the field, this book will appeal to the growing international body of scholars concerned with studying and theorizing about the professions.
Article
Building on the concepts of professional competence that he introduced in his classic The Reflective Practitioner, Schon offers an approach for educating professional in all areas that will prepare them to handle the complex and unpredictable problems of actual practice with confidence, skill, and care.
Article
Valid self-assessment is fundamental to continuing professional competence but is seldom explicitly taught in health professions training. This review analyzed 18 scholarly articles published between January 1970 and February 1990 (14 articles regarding health professions trainees, and four concerning college students or graduate trainees) in which it was possible to compare performance as self-assessed by trainees with performance as assessed by experts or objective tests. The validity of self-assessed performance was found to be low to moderate and did not improve with time in conventional health professions training programs. Self-assessed performance seemed closely related to generalized self-attributions and was minimally influenced by external feedback in the form of test scores, grades, or faculty assessments. In five programs emphasizing explicit self-assessment goals and training strategies, moderate-to-high validity outcomes or improvements over time were demonstrated. Much of what passes for self-assessment in training seems the exercise of an underdeveloped skill, but effective training to improve validity and accuracy is available and feasible.
Article
This study examined how 360 degree feedback ratings and self-other rating discrepancies related to reactions to feedback, perceptions of feedback accuracy, perceived usefulness of the feedback, and recipients' receptivity to development. The results indicated that less favorable ratings were related to beliefs that feedback was less accurate and to negative reactions. Negative reactions and perceptions that feedback was less accurate were related to beliefs that the feedback was less useful. Those who found feedback less useful were perceived by a facilitator as less development-focused. Goal orientation did not moderate the relationship between ratings and perceptions of accuracy or reactions to feedback. Goal orientation was related to perceptions of usefulness of the process several weeks after receipt of feedback. The results question widely held assumptions about 360 degree feedback that negative and discrepant feedback motivates positive change.
Article
To describe responses of family physicians, their medical colleagues, and coworker raters to a multisource feedback assessment process. Data collection tools included multisource feedback self-assessment and medical colleague, coworker, and patient rating forms; and program evaluation physician and rater questionnaires. The pilot study included 142 physicians and their raters, with 113 (80%) physicians completing evaluations. Positive correlations were found between familiarity scores and medical colleague and coworker mean ratings. Peer medical colleagues were significantly more familiar with physicians than were consultants. Consultants were unable to rate items most frequently. Physicians disagreed with colleague feedback more frequently. Agreement was positively correlated with scores. Familiarity, ability to observe physicians appropriately to rate them, and physicians' responses to feedback are factors to consider when multisource feedback is used.
Article
The capability to reflect consciously upon one's professional practice is generally considered important for the development of expertise and, hence, for education. However, to our knowledge no empirical research has been conducted to date into the nature of reflective practice in medicine. To study the structure of reflective practice in medicine. A questionnaire based on the literature was developed and administered to a group of primary care doctors. The data were subjected to confirmatory factor analysis using structural equations modelling. A 5-factor model of reflective practice emerged. It consisted of the following factors: deliberate induction; deliberate deduction; testing and synthesising; openness for reflection, and meta-reasoning. The model fitted the data sufficiently. A multidimensional structure of reflective practice in medicine was brought to light by the study. Its components in terms of reasoning processes, behaviours and attitudes were identified and measured among doctors. Once conceptualised and measured, reflective practice can be studied to gain a better understanding of its relation to expertise development in medicine. In addition, training students to apply reflective practices may become a goal in medical education.
Article
Previous research has demonstrated self-assessment inaccuracy in medical students. This study aims to examine students' and assessors' attitudes towards students' self-assessment of personal and professional behaviours. Twenty-eight [corrected] participants (16 [corrected] Year 1 medical students and 12 personal and professional development assessors) participated in 4 semistructured focus group discussions in April and May 2003. All discussions were audio-taped and transcribed verbatim and the transcripts were theme analysed independently by 2 analysts. Assessors and students perceived accurate self-assessment to be difficult for students and feedback was deemed to be crucial in helping students develop accurate self-assessment of their personal and professional behaviours. Assessors thought that some students had unrealistically high expectations of their own performance and this was thought to be due to various factors, such as previous academic success and gender. Assessors felt that students with high expectations of their own performance exhibited difficult behaviours if they failed to achieve their expectations. Students suggested that the school and the assessors had too high a level of expectation of their personal and professional behaviours, leading them to underestimate students' performance. These difficulties surrounding self-assessment accuracy support the findings reported in previous literature and suggest that medical educators should encourage students to self-assess their own performance wherever possible. These results need to be triangulated with other sources of data such as expert panels or quantitative data.
Article
Physician performance is comprised of several domains of professional competence. Multisource feedback (MSF) or 360-degree feedback is an approach used to assess these, particularly the humanistic and relational competencies. Research studying responses to performance assessment shows that reactions vary and can influence how performance feedback is used. Improvement does not always result, especially when feedback is perceived as negative. This small qualitative study undertook preliminary exploration of physicians' reactions to MSF, and perceptions influencing these and the acceptance and use of their feedback. We held focus groups with 15 family physicians participating in an MSF pilot study. Qualitative analyses included content and constant comparative analyses. Participants agreed that the purpose of MSF assessment should be to enhance practice and generally agreed with their patients' feedback. However, responses to medical colleague and co-worker feedback ranged from positive to negative. Several participants who responded negatively did not agree with their feedback nor were inclined to use it for practice improvement. Reactions were influenced by perceptions of accuracy, credibility and usefulness of feedback. Factors shaping these perceptions included: recruiting credible reviewers, ability of reviewers to make objective assessments, use of the assessment tool and specificity of the feedback. Physicians' perceptions of the MSF process and feedback can influence how and if they use the feedback for practice improvement. These findings are important, raising the concern that feedback perceived as negative and not useful will have no or negative results, and highlight questions for further study.
Article
The ability of physicians to critically reflect on their professional practice has been increasingly valued. Previous research brought to light the multidimensional structure of reflective practice in medicine. It comprises at least five sets of behaviours in response to complex medical problems encountered in professional practice. Factors associated to reflective practice among physicians have, as far as we know, not yet been explored by empirical study. To study factors correlated to reflective practice among physicians. A questionnaire exploring characteristics of professional practice and educational experiences was administered to primary health care physicians. Measurements were related to scores on a reflective practice measuring instrument developed previously. Associations between variables were examined by statistical analysis with tests of correlation and analysis of variance. Reflective practice is negatively correlated to physician's age and number of years of clinical practice. Working mainly in hospitals and attendance to medical residency programmes in some specialties apparently have a positive effect on reflective practice. Reflective practice tends to decrease with experience. Findings are consistent with the literature on medical expertise that shows a decline of analytical reasoning in proportion to the increase in experience. Some specialty programmes seems to enhance concerns with the scientific basis to professional practice, thereby favouring reflective approaches. Local features of primary health care settings probably explain their negative effect on reflective practice. Strategies to develop reflective practice among physicians should be explored by further research.