Article

How Can I Know What I Don't Know? Poor Self Assessment in a Well-Defined Domain

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

As the rapidity with which medical knowledge is generated and disseminated becomes amplified, an increasing emphasis has been placed on the need for physicians to develop the skills necessary for life-long learning. One such skill is the ability to evaluate one's own deficiencies. A ubiquitous finding in the study of self-assessment, however, is that self-ratings are poorly correlated with other performance measures. Still, many educators view the ability to recognize and communicate one's deficiencies as an important component of adult learning. As a result, two studies have been performed in an attempt to improve upon this status quo. First, we tried to re-define the limits within which self-assessments should be used, using Rosenblit and Keil's argument that calibration between perceived and actual performance will be better within taxonomies that are regularly tested (e.g., factual knowledge) compared to those that are not (e.g., conceptual knowledge). Second, we tried to norm reference individuals based on both the performance of their colleagues and their own historical performance on McMaster's Personal Progress Inventory (a multiple choice question test of medical knowledge). While it appears that students are able to (a) make macro-level self-assessments (i.e., to recognize that third year students typically outperform first year students), and (b) judge their performance relatively accurately after the fact, students were unable to predict the percentage of questions they would answer correctly with a testing procedure in which they have had a substantial amount of feedback. Previous test score was a much better predictor of current test performance than were individuals' expectations.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Both (?) tutor and tutoring are important factors which influence PBL process and learning outcomes [10]. Though tutors are in a better position to assess students' skills and abilities during the PBL process, several studies highlighted the difficulty in generating reliable ratings of the tutors [11][12][13][14]. The outcome of tutors' evaluation of students in PBL tutorials has been contentious in terms of the validity of the ratings and scores given to different students [10][11][12][13][14]. Similar 'hawk-dove' effect has been observed in clinical examination where examiners differ in their relative leniency or stringency [15]. ...
... Though tutors are in a better position to assess students' skills and abilities during the PBL process, several studies highlighted the difficulty in generating reliable ratings of the tutors [11][12][13][14]. The outcome of tutors' evaluation of students in PBL tutorials has been contentious in terms of the validity of the ratings and scores given to different students [10][11][12][13][14]. Similar 'hawk-dove' effect has been observed in clinical examination where examiners differ in their relative leniency or stringency [15]. ...
... This may be due to the absence of an 'effective statistical technique' to examine it [15]. Well trained tutors using well-constructed rubrics may eliminate these discrepancies [11][12][13]18]. ...
Article
Full-text available
Background Ensuring objectivity and maintaining reliability are necessary in order to consider any form of assessment valid. Evaluation of students in Problem-Based Learning (PBL) tutorials by the tutors has drawn the attention of critiques citing many challenges and limitations. The aim of this study was to determine the extent of tutor variability in assessing the PBL process in the Faculty of Medical Sciences, The University of the West Indies, St Augustine Campus, Trinidad and Tobago. Method All 181 students of year 3 MBBS were assigned randomly to 14 PBL groups. Out of 18 tutors, 12 had an opportunity to assess three groups: one assessed 2 groups and 4 tutors assessed one group each; at the end each group had been assessed three times by different tutors. The tutors used a PBL assessment rating scale of 12 different criteria on a six-point scale to assess each PBL Group. To test the stated hypotheses, independent t-test, one-way ANOVA followed by post-hoc Bonferroni test, Intra Class Correlation, and Pearson product moment correlations were performed. Result The analysis revealed significant differences between the highest- and lowest-rated groups (t-ratio = 12.64; p < 0.05) and between the most lenient and most stringent raters (t-ratio = 27.96; p < 0.05). ANOVA and post-hoc analysis for highest and lowest rated groups revealed that lenient- and stringent-raters significantly contribute (p < 0.01) in diluting the score in their respective category. The intra class correlations (ICC) among rating of different tutors for different groups showed low agreement among various ratings except three groups (Groups 6, 8 and 13) (r = 0.40). The correlation between tutors’ PBL experiences and their mean ratings was found to be moderately significant (r = 0.52; p > 0.05). Conclusion Leniency and stringency factors amongst raters affect objectivity and reliability to a great extent as is evident from the present study. Thus, more rigorous training in the areas of principles of assessment for the tutors are recommended. Moreover, putting that knowledge into practice to overcome the leniency and stringency factors is essential.
... Pierce and Smith (2001) referred to this increase in accuracy as the postdiction superiority effect. For example, research investigating exams in the classroom (across diverse student populations) has demonstrated that the accuracy of global judgments is better for postdictions than predictions (e.g., Bol & Hacker, 2001;Bol et al., 2005;Bol, Hacker, Walck, & Nunnery, 2012;DiGiacomo & Chen, 2016;Eva, Cunnington, Reiter, Keane, & Norman, 2004;Gutierrez & Price, 2017;Hacker et al., 2000). ...
... We focused on students' judgments about their learning of concepts in an undergraduate psychology course (i.e., category learning judgments, henceforth referred to as CLJs). CLJs were introduced by Jacoby, Wahlheim, and Coane (2010), who investigated learners' accuracy in judging their ability to classify new exemplars of a natural category (bird families) and have since been investigated with other materials and in other settings (DeSoto & Votta, 2016;Doyle & Hourihan, 2016;Eva et al., 2004;Hartwig & Dunlosky, 2017;Morehead, Dunlosky, & Foster, 2017;Rawson, Thomas, & Jacoby, 2015;Riggs, 2012;Tauber & Dunlosky, 2015;Thomas, Finn, & Jacoby, 2016;Wahlheim & DeSoto, 2017;Wahlheim, Dunlosky, & Jacoby, 2011;Wahlheim, Finn, & Jacoby, 2012;Yan, Bjork, & Bjork, 2016;Žauhar, Bajšanski, & Domijan, 2016). We use them in a slightly different context here -to discover the extent to which students can accurately judge their knowledge of a group of related ideas (i.e., concepts) presented in classroom material. ...
... Sense of Urgency Vs Actual Urgency Students described feeling a sense of urgency to treat the patient quicker, however this did not align with their actual time to treat and decision making patterns.This suggests that participants' perceptions of performance may not align with actual performance. This phenomenon is not new -Eva et al (2004) demonstrate the correlation between self-assessment and actual performance in students is poor[194] -"How can I know what I don't know?". In a practical sense, authentic portrayal of moulage might provide more opportunities for students to bridge the gap between self-awareness and performance; moulage presents an opportunity to deliver feedback from objective ...
... Sense of Urgency Vs Actual Urgency Students described feeling a sense of urgency to treat the patient quicker, however this did not align with their actual time to treat and decision making patterns.This suggests that participants' perceptions of performance may not align with actual performance. This phenomenon is not new -Eva et al (2004) demonstrate the correlation between self-assessment and actual performance in students is poor[194] -"How can I know what I don't know?". In a practical sense, authentic portrayal of moulage might provide more opportunities for students to bridge the gap between self-awareness and performance; moulage presents an opportunity to deliver feedback from objective ...
Thesis
Moulage in the traditional sense is the art of replicating illnesses and wounds through casting wax moulds. Origins are traced to Ancient Egypt and forbidden practices of 17th century Europe. While traditional moulage is now housed in musea across the world, modern moulage is used to replicate illness and effects in simulation using special effects makeup techniques. Simulation is a well-established technique to prepare health professionals for clinical practice, and is grounded in a strong evidence base. Despite the strong evidence for the use of simulation, the conditions of moulage is an underexplored topic within the context of simulation research, and we know very little regarding how it works, under what conditions and what the effect is on participants of simulation. In order to better understand how and why moulage impacts on participants of simulation, a series of complementary studies were completed. Initially a Systematic Review of authentic moulage in simulation was undertaken to understand the current research on moulage. This provided a useful baseline for the current use and evidence for moulage in simulation. Subsequently, a further study was undertaken to define authentic moulage in simulation via an electronic Delphi consensus method. This study recruited international experts on moulage and resulted in the development of the Moulage Authenticity Rating Scale (MARS) to measure moulage authenticity. Finally, a third study was conducted to explore how the authenticity of moulage effects participant engagement in simulation using a randomized control experiment design. This study utilized the MARS tool developed from the previous study to design moulage that was low-authenticity and high-authenticity, and compared levels of engagement using measures of self-report, eye tracking and interview methods. The results of this work presents previously unrecognized information on how medical students perceive the authenticity of moulage and how it contributes to their performance and engagement in simulation. In summary, I present a number of suggestions as to how simulation users and designers might consider moulage in their everyday practice. This thesis presents a series of philosophical research questions and findings that collectively make an original contribution to the future of moulage in simulation and undergraduate Medical Education using simulation-based curriculum, teaching and learning. http://hdl.handle.net/1959.13/1412570
... However, studies have shown that students' ability to self-assess their competences is limited. 56,57 In keeping with these findings, our study shows that only 35% of students were able to accurately assess their ability to perform a given task. Research indicates, however, that self-efficacy is a key element in SDL, 56 and achieving competent performance is linked to realistic self-appraisal and self-efficacy. ...
... 56,57 In keeping with these findings, our study shows that only 35% of students were able to accurately assess their ability to perform a given task. Research indicates, however, that self-efficacy is a key element in SDL, 56 and achieving competent performance is linked to realistic self-appraisal and self-efficacy. 58 Integrating self-directed simulator-based teaching in the veterinary curriculum should therefore incorporate self-assessment training throughout the course of education in order to achieve optimal results. ...
Article
While the use of simulator-based clinical skill training has become increasingly popular in veterinary education in recent years, little research has been done regarding optimal implementation of such tools to maximize student learning in veterinary curricula. The objective of this study was to compare the effects of supervised and unsupervised deliberate practice on clinical skills development in veterinary medicine students. A total of 150 veterinary students took part in instructor-led practice (supervised) or self-directed practice (unsupervised) at a selection of four learning stations in a veterinary skills laboratory. Each learning station consisted of a teaching simulator, materials required to complete the task, and a standard operating procedure detailing how to execute the task. Students used Likert scales to self-evaluate their clinical skills before and after practice sessions, in addition to evaluating their motivation to practice a given task. An objective structured clinical examination (OSCE) was used to compare participants' clinical skills performance between learning stations. We were able to show that practice had a significant positive effect on OSCE scores at three out of six available learning stations. Motivation ratings varied between learning stations and were positively correlated with an increase in self-perceived clinical skills. At an instructor-to-student ratio of approximately 1:8, supervision had no effect on OSCE scores at four out of six learning stations. At the remaining two learning stations, self-directed practice resulted in better learning outcomes than instructor-led practice.
... 28,62,63 This encouraged self-reflection in students, another important component of self-regulated learning. 23,[64][65][66] Reflection is an important and necessary skill to facilitate or support patient-centered care, enabling the pharmacist to fully comprehend the patients' needs, concerns, and expectations, as well as their own values and feelings. 14 It is also an effective tool for health professionals as by reflecting on their strengths and weaknesses, they are able to determine their own training needs, as well as develop confidence in their professional capabilities. ...
Article
Background and purpose: Pharmacists need to learn communication styles which adopt a more consultative model. The objectives of this paper are to describe the use of forum theatre facilitated by actors to teach communication skills to pharmacy students and to highlight perceptions of the workshops. Educational activity and setting: During the actor-led forum theatre, students redirected scenes depicting pharmacist-patient consultations. Students also learned about their own communication styles and role-played consultations and interview situations with actors. Pharmacy undergraduate students enrolled in years 2, 3, and 4 answered open-ended questions online at the end of workshops. All responses were coded to identify themes. Findings: Five themes emerged from 752 comments: (1) workshops were useful in facilitating communication skills development, (2) workshops encouraged reflection, (3) appreciation of feedback provided, (4) the live, interactive nature of the workshop enhanced learning, and (5) suggestions for improvement. There were requests for more varied scenarios involving different patient groups and more opportunities to participate in the role-plays. Students also requested smaller groups, which would encourage participation from those who were reticent and allow more personal feedback. Summary: The use of forum theatre employing role-play, small group coaching on consultation and interview skills, and emphasizing feedback was perceived as an effective and engaging method to teach communication skills.
... [1] In the near future, medical knowledge will double every few months; therefore, it is critical that medical students develop life-long learning skills [2]. Self-directed learning (SDL) is considered an important component of life-long learning and thus is a key competency in medical school curricula [3]. Historically, self-directed learning was defined by Knowles as a process in which a learner takes the initiative, diagnoses their learning needs, creates learning goals, identifies resources for learning, applies appropriate learning strategies and evaluates their learning outcomes [4]. ...
Article
Full-text available
Background: It is critical that medical students develop self-directed, life-long learning skills to navigate medical school successfully and to become competent healthcare professionals. Moreover, the Liaison Committee on Medical Education (LCME), the USA medical school accrediting body, requires activities designed to help students develop self-directed learning (SDL) skills in the preclinical years. Objective: We evaluated the feasibility and effectiveness of a self-directed learning activity in a 6-week first-year medical student course. Design: The course director assigned infectious disease case studies to teams of first-year medical students who individually assessed their knowledge gaps of the case, identified scholarly sources to fill their knowledge gaps, shared the information with their teammates, and reflected on their ability to guide their own learning. Students were asked to rate workload, team effort, acquisition of new clinical knowledge, and life-long learning skills. Students were also asked to reflect on how this assignment affected their perception of their SDL skills. Descriptive statistics were used to analyze responses to the Likert scale questions. Thematic analysis was applied to the comments. Results: Survey response rate was 80% (131/163). Students strongly or moderately agreed that 1) they spent an appropriate amount of time on the project (94%), 2) the workload was evenly distributed among their teammates (95%), 3) their teammates made significant and timely contributions to the project (97%), 4) the project contributed to learning new clinical knowledge (92%), and 5) the project contributed to the acquisition of life-long learning skills (85%). The analysis team identified four themes from student reflections on their perception of their self-directed learning skills: self-learning skills, collaboration, application, and meta-cognition, Conclusions: Study results demonstrated that we successfully implemented a case-based SDL activity in a first-year medical school course and that students perceived the activity as a valuable learning experience.
... 17 Thus, as overall resources become less available, programs may be forced to prioritize improvements based on a variety of factors including evolving expectations from accrediting bodies, changes in the pharmacy practice environment, as well as shifts in the recognition of the importance of enhancing student and faculty personal well-being and professional development. 18,19 As pharmacy programs aim to provide high quality education experiences while meeting accreditation expectations, more information is needed to better understand successful and sustainable co-curriculum. Given the dynamic environment surrounding accreditation requirements and lack of consistent definitions related to the co-curriculum, our group sought to identify and describe the barriers facing pharmacy programs in implementing the co-curriculum in their programs and evaluate programs' confidence in meeting the new co-curriculum ACPE requirement. ...
Article
Objective. To determine areas of concern, and challenges to implementing and assessing the co-curriculum in accredited Doctor of Pharmacy programs, along with how confident programs are in their ability to meet the co-curriculum requirement as mandated by the Accreditation Council for Pharmacy Education (ACPE). Methods. A survey was administered to all ACPE-accredited pharmacy programs to collect information regarding areas of concern, challenges, and confidence in their ability to meet the co-curriculum requirement. The frequency of responses to items are presented along with comparisons based on characteristics, including institution type, cohort size, most recent ACPE accreditation review, and supporting offices. Results. The most common concerns centered on the documentation and assessment process. The most commonly reported challenges were lack of enthusiasm or buy-in from faculty, staff, and students; lack of a clear definition of co-curriculum; and faculty time and insufficient staff. Overall, programs had a high level of confidence in their ability to meet the requirements for co-curriculum. The only differences found were related to supporting offices and cohort size. Conclusion. The results suggest that having supporting offices may reduce the co-curriculum burden. Similarly, student cohort size may have an impact on the challenges for some programs, particularly those with moderate-sized cohorts reporting challenges related to faculty and staff. Further research is needed to determine how programs address these critical issues, and to explore whether programs report differently on these areas after completing an accreditation review. The study results may be useful to members of the Academy when evaluating co-curriculum.
... Moreover, as demonstrated in this study, self-reported competence is weakly related to performance-based competence outcomes. Consistent with other work on competency, the relationship between self-evaluated competence and performance-based competence is minimal (Davis et al., 2006;Eva et al., 2004). As such, focus on self-awareness and self-knowledge in competency development (Kaslow et al., 2018) would benefit from ensuring trainee perceptions of their competency align with benchmarked progression. ...
Article
Attaining competence in assessment is a necessary step in graduate training and has been defined to include multiple domains of training relevant to this attainment. While important to ensure trainees meet these standards of training, it is critical to understand how and if competence shapes a trainees' professional identity, therein promoting lifelong competency. The current study assessed currently enrolled graduate trainees' knowledge and perception of their capabilities related to assessment to determine if self‐reported and performance‐based competence would incrementally predict their intention to use assessment in their future above basic training characteristics and intended career interests. Self‐reported competence, but not performance‐based competence, played an incremental role in trainees' intention to use assessments in their careers. Multiple graduate training characteristics and practice experiences were insignificant predictors after accounting for other relative predictors (i.e., intended career settings, integrated reports). Findings are discussed about the critical importance of incorporating a hybrid competency‐capability assessment training framework to further emphasize the role of trainee self‐efficacy in hopes of promoting lifelong competence in their continued use of assessments. This study examines health service psychology trainees' intention to engage in psychological assessment and the incremental role of self‐reported competence. Findings highlight the significance of self‐reported competence above performance‐based competence in assessment when predicting long‐term professional identity, which in turn promotes lifelong competence.
... Although the Accreditation Council for Graduate Medical Education recommends that residents complete self-assessments periodically, 4 studies have demonstrated that self-assessments by medical students, interns, and residents in general surgery, family medicine, and psychiatry are invalid measures of their objective performance. [5][6][7][8][9] Several medical studies support different portions of the Dunning-Kruger Curve. For example, 1 study found that junior physicians who were less competent tended to self-rate their efficacy higher than it was demonstrated (point A). 10 Comparison of selfassessments and peer assessments of physicians in internal medicine, pediatrics, and psychiatry showed that those assessed in the lowest quartile by their colleagues tended to rate themselves 30 to 40 percentile ranks higher than their peers. ...
... Am I about to do the right thing?), whilst self-assessment refers to a judgment on one's cumulative evaluation of overall performance (e.g. do I need to refresh my understanding of the best practice for this?) [2,3,5]. ...
Article
Full-text available
Background: Self-monitoring is an important component of clinical practice. It is underpinned by the framework of self-efficacy which is concerned with judgments of how well one believes one can achieve or perform a task. This research aimed to develop criteria for adequate self-monitoring, then to measure patterns of self-monitoring, and to explore how these patterns relate to a student's year in a medical course and to patterns of knowledge. Methods: Analysis of individuals' levels of correctness in answering assessment items and their certainty in correctness may be used to inform assessments of ability to self-monitor. Two criteria were proposed and applied to define adequate self-monitoring. Firstly, increasing proportions correct with increasing levels of certainty. Secondly, having a proportion correct for high certainty responses that was not lower than cohort levels. Student responses in progress tests comprising multiple-choice questions (MCQs) and associated certainty were analysed. Criteria for the presence of adequate self-monitoring and for adequate knowledge were applied to the results of each of four tests conducted over 2 years, and used to categorise patterns of self-monitoring and knowledge. Results: Data from 3 year group cohorts totalling 737 students were analysed. The majority (58%) of students demonstrated adequate knowledge and met both criteria for adequate self-monitoring across all four tests. The most advanced year group cohort had the highest rates of adequate knowledge and the highest rates of meeting both self-monitoring criteria. The patterns of self-monitoring were the same as the patterns of knowledge across the four tests for 454 students, but for the remaining 283 the patterns of self-monitoring and knowledge differed. Conclusion: Analysis of responses to item level certainty has informed development of a definition of adequate self-monitoring that may be applied to individual student's responses from a single test, and to track the adequacy of a student's self-monitoring over time. Patterns of self-monitoring tend to match patterns of knowledge, but not in all cases, suggesting the self-monitoring measure could provide additional information about student ability.
... Last, while self-assessment has been argued to enhance learning in medical education, 101 most of the studies in our scoping review collected data based on learners' self-assessment of confidence as opposed to potentially more objective measures. 102,103 The relationship between a learners' confidence in performing a skill and subsequent performance is significant nonetheless. 104 Confidence in performance influences the willingness and ultimate decision to perform and thus if learner confidence does not match the subsequent performance, the learner may fall short and potentially risk an adverse event. ...
Article
Full-text available
Phenomenon: There is currently a move to provide residency programs with accurate competency-based assessments of their candidates, yet there is a gap in knowledge regarding the role and effectiveness of interventions in easing the transition to residency. The impact of key stakeholder engagement, learner-centeredness, intrinsic competencies, and assessment on the efficacy of this process has not been examined. The objective of this scoping review was to explore the nature of the existing scholarship on programs that aim to facilitate the transition from medical school to residency. Approach: We searched MEDLINE and EMBASE from inception to April 2020. Programs were included if they were aimed at medical students completing undergraduate medical training or first year residents and an evaluative component. Two authors independently screened all abstracts and full text articles in duplicate. Data were extracted and categorized by type of program, study design, learner-centeredness, key stakeholder engagement, the extent of information sharing about the learner to facilitate the transition to residency, and specific program elements including participants, and program outcomes. We also extracted data on intrinsic (non-Medical Expert) competencies, as defined by the CanMEDS competency framework. Findings: Of the 1,006 studies identified, 55 met the criteria for inclusion in this review. The majority of the articles that were eligible for inclusion were from the United States (n = 31, 57%). Most of the studies (n = 47, 85%) employed quantitative, or mixed method research designs. Positive outcomes that were commonly reported included increased self-confidence, competence in being prepared for residency, and satisfaction with the transition program. While a variety of learner-centered programs that focus on specific intrinsic competencies have been implemented, many (n = 29, 52%) did not report engaging learners as key stakeholders in program development. Insights: While programs that aim to ease the transition from medical school to residency can enhance both Medical Expert and other intrinsic competencies, there is much room for novel transition programs to define their goals more broadly and to incorporate multiple areas of professional development. The existing literature highlights various gaps in approaches to easing the transition from medical school to residency, particularly with respect to key stakeholder engagement, addressing intrinsic CanMEDS competencies, and focusing on individual learners' needs.
... Previous studies have found that improving the skills of participants in social and intellectual domains paradoxically helps them recognize the limitations of their abilities 23 . The OSCE provides both a tool for self-assessment, helping A c c e p t e d M a n u s c r i p t 12 medical trainees recognize the deficits that exist, as well as the opportunity to rectify this deficit through immediate feedback and discussion 24 . ...
Article
Full-text available
Purpose Transgender and gender non-conforming individuals have unique healthcare needs and have difficulty accessing healthcare services due to lack of qualified healthcare providers, insurance coverage, mistreatment and bias by the medical community. Medical trainees and physicians report a lack of education in, and exposure to, the clinical care and unique aspects of this field. We assessed the use of a standardized patient as a tool to evaluate four core medical competencies (patient care, medical knowledge, professionalism and interpersonal communication) of endocrinology fellows at a single training program. Methods Endocrine fellows were evaluated by faculty in different aspects of transgender care and completed a self-assessment before and after the exercise. Faculty viewed the fellows during the OSCE examination. Fellows were provided feedback by a faculty member and the standardized patient after the exercise. Results Deficits were found in patient care and professionalism. Fellows scored well in medical knowledge. Fellows did not report an improvement in comfort and communication skills after the exercise. Interestingly, fellows’ self-assessment scores in several domains declined after the standardized patient encounter, highlighting an occasion for self-reflection and growth within the realms of cultural competency and medical knowledge. Main conclusions We conclude that use of standardized patients to teach medical competencies in transgender medicine may be one approach to improve exposure to, and training in, transgender medicine. Endocrine fellows still had discomfort with treating transgender individuals after the standardized patient encounter and require other training activities which may include didactics and clinical case discussions.
... We used a proxy measure of behavior change in this study: students' self-reported confidence and feelings of preparedness to apply the skills they learned in future patient scenarios. Feelings of self-efficacy are felt to correlate with increased likelihood of success in applying the skills one has learned [36][37][38]. In our study, students had the opportunity to reflect on the learning experience (CS-M and CS-NM, each with facilitated debriefing) before responding to questions about their confidence and preparedness. ...
Article
Context Situativity theory posits that learning and the development of clinical reasoning skills are grounded in context. In case-based teaching, this context comes from recreating the clinical environment, through emulation, as with manikins, or description. In this study, we sought to understand the difference in student clinical reasoning abilities after facilitated patient case scenarios with or without a manikin. Methods Fourth-year medical students in an internship readiness course were randomized into patient case scenarios without manikin (control group) and with manikin (intervention group) for a chest pain session. The control and intervention groups had identical student-led case progression and faculty debriefing objectives. Clinical reasoning skills were assessed after the session using a 64-question script concordance test (SCT). The test was developed and piloted prior to administration. Hospitalist and emergency medicine faculty responses on the test items served as the expert standard for scoring. Results Ninety-six students were randomized to case-based sessions with (n = 48) or without (n = 48) manikin. Ninety students completed the SCT (with manikin n = 45, without manikin n = 45). A statistically significant mean difference on test performance between the two groups was found (t = 3.059, df = 88, p = .003), with the manikin group achieving higher SCT scores. Conclusion Use of a manikin in simulated patient case discussion significantly improves students’ clinical reasoning skills, as measured by SCT. These results suggest that using a manikin to simulate a patient scenario situates learning, thereby enhancing skill development.
... The assessment was driven by the demand for constructive adjustment between intentions and evaluations of PBL, the conventional summative assessment model was not really t well to PBL [15]. Hence, the formative assessment protocols including student performance, student self-mutual evaluation and tutor teaching evaluation were adopted in this study, which provide unique opportunity to judge each other's work between tutors and students wherefore making them develop the ability to re ect on own strengths and weaknesses as these are central elements of self-directed learning (SDL) [16]. ...
Preprint
Full-text available
Objectives: Evaluate the effectiveness of PBL teaching and analyze potential influencing factors between senior and junior from the perspectives of student performance, student self-mutual and tutor teaching evaluations. Results: Study found the overall three comparative assessments scores of Grade 4 were significantly higher than those of Grade 1(P<0.01). Further analysis revealed no significant difference of student performance score between Grade 4 and Grade 1 in the 1st unit (P>0.05), but it was improved and maintained significantly since 2nd time (P<0.01).Moreover, each self-mutual evaluation of 4th year students was significantly higher than that of 1st year students, especially in the second half of the course (P<0.01).And, all the evaluations of tutor teaching from seniors were significantly higher than those from juniors (P<0.01). Finally, no significant correlation between student self-mutual scores and those given by tutors no matter high or lower grade (P>0.05).
... Again, the concept of self-regulation of learning in the educational setting is not undisputed. For example, the ability of students to successfully self assess and subsequently direct their own learning is seriously doubted (Eva et al. 2004). Yet, there seems to be more agreement that after graduation doctors should be able to be lifelong learners and for this require having developed self-assessment and self-regulated learning ability. ...
Article
Full-text available
Arguably, constructive alignment has been the major challenge for assessment in the context of problem-based learning (PBL). PBL focuses on promoting abilities such as clinical reasoning, team skills and metacognition. PBL also aims to foster self-directed learning and deep learning as opposed to rote learning. This has incentivized researchers in assessment to find possible solutions. Originally, these solutions were sought in developing the right instruments to measure these PBL-related skills. The search for these instruments has been accelerated by the emergence of competency-based education. With competency-based education assessment moved away from purely standardized testing, relying more heavily on professional judgment of complex skills. Valuable lessons have been learned that are directly relevant for assessment in PBL. Later, solutions were sought in the development of new assessment strategies, initially again with individual instruments such as progress testing, but later through a more holistic approach to the assessment program as a whole. Programmatic assessment is such an integral approach to assessment. It focuses on optimizing learning through assessment, while at the same gathering rich information that can be used for rigorous decision-making about learner progression. Programmatic assessment comes very close to achieving the desired constructive alignment with PBL, but its wide adoption—just like PBL—will take many years ahead of us.
... Our findings about the aspects of podcasts that were most influential or impactful for residents' learning thus should be contextualized within the vast literature on the limitations of self-assessment. 45,46 That said, several participants did seem to describe moments of self-monitoring and self-regulation; podcasts therefore may provide a useful medium for future explorations into how trainees make in-the-moment judgments about content alignment and learning goals. 47,48 Conclusions Despite the challenges of distracted, contextually constrained listening and difficulties translating their learning into clinical practice, residents found podcasts to be an accessible and engaging learning platform that offered them broad exposure to core content and personalized learning, concurrently fostering their sense of connection to local and national professional communities. ...
Article
Purpose: Educational podcasts are an increasingly popular platform for teaching and learning in health professions education. Yet it remains unclear why residents are drawn to podcasts for educational purposes, how they integrate podcasts into their broader learning experiences, and what challenges they face when using podcasts to learn. Method: The authors used a constructivist grounded theory approach to explore residents' motivations and listening behaviors. They conducted 16 semi-structured interviews with residents from 2 US and 1 Canadian institution from March 2016 to August 2017. Interviews were recorded and transcribed. The transcripts were analyzed using constant comparison and themes were identified iteratively, working toward an explanatory framework that illuminated relationships among themes. Results: Participants described podcasts as easy to use and engaging, enabling both broad exposure to content and targeted learning. They reported often listening to podcasts while doing other activities, being motivated by an ever-present desire to use their time productively; this practice led to challenges retaining and applying the content they learned from the podcasts to their clinical work. Listening to podcasts also fostered participants' sense of connection to their peers, supervisors, and the larger professional community, yet it created tensions in their local relationships. Conclusions: Despite the challenges of distracted, contextually-constrained listening and difficulties translating their learning into clinical practice, residents found podcasts to be an accessible and engaging learning platform that offered them broad exposure to core content and personalized learning, concurrently fostering their sense of connection to local and national professional communities.
... The low correlations between the scales of the self-assessment indicate that the students considered a different frame of reference when assessing their performance in the conversation. Eva, Cunnington, Reiter, Keane and Norman (2004) suggested that evaluations of our own ability are biased because we are unable to remember every detail of our behaviour correctly. Furthermore, standardised patients scored higher compared to the other two measures. ...
Article
This work investigates the German version of the Kalamazoo Communication Skills Assessment Form (KCSAFd) for three assessment methods: students' self-assessment (KCSAFd-self), assessment by standardised patients (KCSAFd-sPat) and video-assessment by trained raters (KCSAFd-video). Videotaped simulated patient consultations of N = 163 medical students from the first (n = 97) and the final clinical years (n = 66) were rated using the KCSAFd. Investigating the psychometric properties of the instrument, we found a two factor-construct with interpersonal and conversational competence. All methods showed good internal consistency and acceptable model fit values. Additionally, we found plausible relationships between the three methods and meaningful differences between the two groups of students.
... [21][22][23] 6 Unlike students or residents who must follow a defined curriculum, independently practicing physicians choose their continuing medical education (CME) activities themselves, relying on self-assessment. 5,[24][25][26][27][28] However, this self-assessment may not be accurate or comprehensive; 29,30 in fact, research shows that those with the lowest ability are the least accurate in assessing their own knowledge and skills. 31 Practicing physicians are sometimes skeptical or resistant toward ...
Article
Full-text available
Purpose: Little is known about how board-certified physicians prepare for their periodic Maintenance of Certification (MOC) examinations. This qualitative study explores how physicians experience MOC exam preparation: how they prepare for the exams and decide what to study and how exam preparation compares with what they normally do to keep their medical knowledge current. Method: Between September 2016 and March 2017, the authors interviewed 80 primary care physicians who had recently taken either the American Board of Family Medicine or American Board of Internal Medicine MOC exam. They analyzed transcripts and notes from these interviews looking for patterns and emergent themes, using the constant comparative method and a social practice theory perspective. Results: Most interviewees studied for their MOC exams by varying from their routines for staying current with medical knowledge, both by engaging with a different scope of information and by adopting different study methods. Physicians described exam preparation as returning to a student/testing mindset, which some welcomed and others experienced negatively or with ambivalence. Conclusions: What physicians choose to study bounds what they can learn from the MOC exam process, and therefore also bounds potential improvements to their patient care. Knowing how physicians actually prepare, and how these preparation activities compare with what they do when not preparing for an exam, can inform debates over the value of requiring such exams, as well as conversations about how physicians, certification boards, and other key stakeholders in physicians' continuing professional development could improve the MOC process.
... A number of studies have reported that females perform better than males in observed examinations although their self-assessment scores tended to be lower [24,33]. There is limited evidence to suggest that students improve their self-assessment skill over time [21,33,36,37], though self-assessment is considered a learnable skill [38]. ...
Article
Full-text available
Understanding our learning needs is fundamental for safe, effective and knowledge-based medical practice and facilitates life-long learning. A mixed methods study investigated fourth-year medical students’ self-perceived understanding of their learning needs using 1] a visual scale, before and after a four-week module in Ageing and Health (A&H) and 2] through focus group discussions. During 2013–14 academic year, all students (252) were invited to use a Visual Analogue Scale (VAS) tool to self-assess their learning needs that were linked to Ageing and Health curriculum learning outcomes. Assenting students (197 at pre-self-assessment, 201 at post-assessment) returned anonymous Visual Analogue Scales, self-assessing history-taking skills, examination skills, knowledge of medication use, co-morbidity, nutritional and swallowing assessment responses, before and after the A&H module. Three student focus groups explored whether completion of the VAS self-assessment had prompted improved self-awareness of their learning needs. The VAS responses increased for each curriculum domain with significant differences between the pre-and post responses – for the student-year-group. Nutritional and swallowing knowledge showed the greatest improvement from a self-assessed low baseline at entry. Focus-group students generally viewed the VAS tool positively, and as an aid for prompting consideration of current and future clinical practice. Some students recognised that ‘a need to be ready-for-work’ focused engaged learning; others demonstrated self-regulated learning through self-motivation and an action plan. The Visual Analogue Scale quantitative responses showed increased student-self-perceived learning for each curriculum domain at fourth-year completion of the A&H module, suggesting that prompting self-assessment had increased students’ knowledge and skills. Focus group students saw the VAS tool as useful for prompting awareness of their current and future learning needs. Additional educational strategies should be explored to enable all students to self-reflect and engage effectively on their learning needs, to gain the skills for the maintenance of professional medical competence. Abbreviations: A&H: Ageing and Health Module; e-portfolio: an electronic version of an evidence portfolio, which allows medical students and graduates to reflect and document learning and competencies; F1: year1 of post-graduate medical clinical training; GMC: General Medical Council-the regulation organisation for maintaining standards for doctors in UK; Logbook: usually a written document which can be used to record procedures and attendance at clinics or case-based discussions and can be used to set learning outcomes and to structure teaching in clinical settings for medical students and doctors; PDP: personal development plan is used to plan future learning and skills needs for work and education with an plan for action/s outcome; SPSS: Statistical Package for the Social Sciences; VAS: Visual Analogue Scale is a visual method of describing an experience.
... Our representations of residents' perceptions of effectiveness must be contextualized within the literature on the limitations of selfassessment and judgments of learning. [50][51][52] These findings should serve as a guide to future realist evaluations that examines these features to determine what works, where, and why, 53 rather than be viewed as a definitive prescription for podcast optimization. ...
Article
Introduction: Educational podcasts are used by emergency medicine (EM) trainees to supplement clinical learning and to foster a sense of connection to broader physician communities. Yet residents report difficulties remembering what they learned from listening, and the features of podcasts that residents find most effective for learning remain poorly understood. Therefore, we sought to explore residents' perceptions of the design features of educational podcasts that they felt most effectively promoted learning. Methods: We used a qualitative approach to explore EM trainees' experiences with educational podcasts, focusing on design features that they found beneficial to their learning. We conducted 16 semi-structured interviews with residents from three institutions from March 2016-August 2017. Interview transcripts were analyzed line-by-line using constant comparison and organized into focused codes, conceptual categories, and then key themes. Results: The five canons of classical rhetoric provided a framework for thematically grouping the disparate features of podcasts that residents reported enhanced their learning. Specifically, they reported valuing the following: 1) Invention: clinically relevant material presented from multiple perspectives with explicit learning points; 2) Arrangement: efficient communication; 3) Style: narrative incorporating humor and storytelling; 4) Memory: repetition of key content; and 5) Delivery: short episodes with good production quality. Conclusion: This exploratory study describes features that residents perceived as effective for learning from educational podcasts and provides foundational guidance for ongoing research into the most effective ways to structure medical education podcasts.
... Most important, a belief in self-efficacy is not itself a measure of effectiveness. Research has shown, repeatedly and across disciplines, that self-assessments inflate communication competence relative to external evaluation (e.g., Duran & Zakahi, 1987;Eva et al., 2004;Gruppen et al., 1997;McCroskey & McCroskey, 1988;Mort & Hansen, 2010). The key measure of the effectiveness of any form of communication training is not only evidence that a target audience judges the trainee effective (Bray, 2012;Rodgers et al., 2018) but also that the target audience finds the trainee a more effective communicator after training than before. ...
Article
Full-text available
As the science community has recognized the vital role of communicating to the public, science communication training has proliferated. The development of rigorous, comparable approaches to assessment of training has not kept pace. We conducted a fully controlled experiment using a semester-long science communication course, and audience assessment of communicator performance. Evaluators scored the communication competence of trainees and their matched, untrained controls, before and after training. Bayesian analysis of the data showed very small gains in communication skills of trainees, and no difference from untrained controls. High variance in scores suggests little agreement on what constitutes “good” communication.
... One benefit of this change is that experienced clinicians often recognize similar situations (e.g., familiar problems) more quickly [68]. Importantly, experience may not improve their ability to adapt commonly used strategies to resolve new problems or increase their ability to recognize errors in their own thinking processes [50,[69][70][71]. Since heuristics will always have a role in decision-making due to the speed expected in real-time interactions, it is important to understand what information is used in heuristic decision-making. ...
Preprint
Full-text available
Significant efforts in the past decades to teach evidence-based practice (EBP) implementation has emphasized increasing knowledge of EBP and developing interventions to support adoption to practice. These efforts have resulted in only limited sustained improvements in the daily use of evidence-based interventions in clinical practice in most health professions. Many new interven-tions with limited evidence of effectiveness are readily adopted each year - indicating openness to change is not the problem. The selection of an intervention is the outcome of an elaborate and complex cognitive process which is shaped by how they represent the problem in their mind and is mostly invisible processes to others. Therefore, the complex thinking process which support appropriate adoption of interventions should be taught more explicitly. Making the process visible to clinicians increases the acquisition of the skills required to judiciously select one in-tervention over others. The purpose of this paper is to provide a review of the selection process and the critical analysis that is required to appropriately decide to trial or not trial new intervention strategies with patients.
... These were designed for a specific skill or purpose with more defined instruction compared to SDL directed at a wider-focused subject. SDL is considered an important component of life-long learning and is a key competency in medical school curricula (Eva et al., 2004). However relevant skills for SDL should be taught to ensure participants can successfully engage with these activities rather than assuming they are inherent (Gaines et al., 2018). ...
Article
Full-text available
Introduction As traditional clinical teaching faces major obstacles during the COVID-19 pandemic, medical educators look toward remote teaching methods to provide solutions to allow continuation of teaching. Remote methods, teaching delivered other than face-to-face, align with the transformation seen within pedagogy over the last 20 years. Aim The aim of this scoping review was to i) identify existing teaching methods available to remotely teach cardiovascular knowledge or skills and ii) identify if they have been evaluated. Methods A scoping review of the literature was undertaken to synthesise available evidence and examine remote teaching methods for application to undergraduate medical education. Results Forty-two articles were identified which presented remote teaching methods using either teaching based online, computer-programs, digital resources, mobile-phone technology, podcasts, serious gaming, social media or resources to aid self-directed learning. Although results were heterogenous, they gave an indication of the method’s usefulness. However, evaluations were not consistent and if they were, would have strengthened the value of the findings. Conclusion Various remote teaching methods are available to replace face-to-face cardiovascular teaching where this is not possible. Evidence for effectiveness and engagement of individual platforms are variable. Given the ongoing COVID-19 pandemic, medical educators should prioritise ongoing evaluation of remote teaching methods and share best practice.
... One benefit of this change is that experienced clinicians often recognize similar situations (e.g., familiar problems) more quickly [68]. Importantly, experience may not improve their ability to adapt commonly used strategies to resolve new problems or increase their ability to recognize errors in their own thinking processes [50,[69][70][71]. Since heuristics will always have a role in decision-making due to the speed expected in real-time interactions, it is important to understand what information is used in heuristic decision-making. ...
Article
Full-text available
Significant efforts in the past decades to teach evidence-based practice (EBP) implementation has emphasized increasing knowledge of EBP and developing interventions to support adoption to practice. These efforts have resulted in only limited sustained improvements in the daily use of evidence-based interventions in clinical practice in most health professions. Many new interventions with limited evidence of effectiveness are readily adopted each year—indicating openness to change is not the problem. The selection of an intervention is the outcome of an elaborate and complex cognitive process, which is shaped by how they represent the problem in their mind and is mostly invisible processes to others. Therefore, the complex thinking process that supports the appropriate adoption of interventions should be taught more explicitly. Making the process visible to clinicians increases the acquisition of the skills required to judiciously select one intervention over others. The purpose of this paper is to provide a review of the selection process and the critical analysis that is required to appropriately decide to trial or not trial new intervention strategies with patients.
... 23,24 The finding that residents were significantly less confident in their learning while driving, despite no difference in assessment of recall, is consistent with the known limitations of self-assessment and judgment of learning. [25][26][27] Residents should be aware that their perceptions of where learning best takes place may not be accurate. Further, they should be aware that recall of podcast knowledge over time is significantly impaired (>20% drop in test scores after 1 month). ...
Article
Study objective Emergency medicine residents use podcasts as part of their learning process, often listening while driving. It is unclear how driving while listening to a podcast affects knowledge acquisition and retention. This study evaluated the knowledge gained from listening to podcasts while driving compared to that gained from undistracted listening. Methods This was a multicenter, randomized, crossover trial among postgraduate year (PGY) 1 to 4 emergency medicine residents at 4 institutions. Residents were randomized with stratification by site and PGY level to listen to podcasts while driving first or sitting undistracted in a room first. Within 30 minutes of listening, they completed a 20-question test. They subsequently crossed over to the alternate intervention, serving as their own controls, and listened to a different podcast before completing a second 20-question test. Each of the podcasts was professionally recorded and based on 5 emergency medicine-relevant journal articles that had not been covered in a journal club or curriculum at any of the institutions. One month later, participants completed a delayed recall test composed of 40 new questions based on both podcasts. Questions were derived and validity evidence was collected prior to use. Data were compared using a paired-sample t test and ANOVA. Results A total of 100 residents completed the initial recall tests, and 96 residents completed the delayed recall test. There was no statistically significant difference between the driving and undistracted cohorts on the initial recall (74.2% versus 73.3%) or delayed recall (52.2% versus 52.0%). Conclusion Driving while listening to a podcast does not meaningfully affect knowledge acquisition or retention when compared with undistracted podcast listening among emergency medicine residents.
... Therefore, our participants' narratives about the aspects of training that were most beneficial for their learning should be contextualized within the literature on the limitations of self-assessment. 40 ...
Article
Full-text available
Introduction: Effective leadership improves patient care during medical and trauma resuscitations. While dedicated training programs can improve leadership in trauma resuscitation, we have a limited understanding of the optimal training methods. Our objective was to explore learners' and teachers' perceptions of effective methods of leadership training for trauma resuscitation. Methods: We performed a qualitative exploration of learner and teacher perceptions of leadership training methods using a modified grounded theory approach. We interviewed 28 participants, including attending physicians, residents, fellows, and nurses who regularly participated in trauma team activations. We then analyzed transcripts in an iterative manner to form codes, identify themes, and explore relationships between themes. Results: Based on interviewees' perceptions, we identified seven methods used to train leadership in trauma resuscitation: reflection; feedback; hands-on learning; role modeling; simulation; group reflection; and didactic. We also identified three major themes in perceived best practices in training leaders in trauma resuscitation: formal vs informal curriculum; training techniques for novice vs more senior learner; and interprofessional training. Participants felt that informal training methods were the most important part of training, and that a significant part of a training program for leaders in trauma resuscitation should use informal methods. Learners who were earlier in their training preferred more supervision and guidance, while learners who were more advanced in their training preferred a greater degree of autonomy. Finally, participants believed leadership training for trauma resuscitation should be multidisciplinary and interprofessional. Conclusion: We identified several important themes for training leaders in trauma resuscitation, including using a variety of different training methods, adapting the methods used based on the learner's level of training, and incorporating opportunities for multidisciplinary and interprofessional training. More research is needed to determine the optimal balance of informal and formal training, how to standardize and increase consistency in informal training, and the optimal way to incorporate multidisciplinary and interprofessional learning into a leadership in trauma resuscitation training program.
... While medical students themselves recognize the importance of science-based decision-making, they often do not feel confident in their ability to do so (Pruskil et al., 2009) although there is also evidence of overconfidence effects (Borracci and Arribalzaga, 2018). This, in turn, may be due to deficits in their ability to evaluate their own knowledge and recognize deficiencies therein (Kruger and Dunning, 1999;Eva et al., 2004). With regard to the learning progressions of medical students, making professional decisions and solving problems in clinical contexts require not only the acquisition of knowledge but also an increase in confidence in their medical knowledge (Khan et al., 2001). ...
Article
Full-text available
Research Focus: The promotion of domain-specific knowledge is a central goal of higher education and, in the field of medicine, it is particularly essential to promote global health. Domain-specific knowledge on its own is not exhaustive; confidence regarding the factual truth of this knowledge content is also required. An increase in both knowledge and confidence is considered a necessary prerequisite for making professional decisions in the clinical context. Especially the knowledge of human physiology is fundamental and simultaneously critical to medical decision-making. However, numerous studies have shown difficulties in understanding and misconceptions in this area of knowledge. Therefore, we investigate (i) how preclinical medical students acquire knowledge in physiology over the course of their studies and simultaneously gain confidence in the correctness of this knowledge as well as (ii) the interrelations between these variables, and (iii) how they affect the development of domain-specific knowledge. Method: In a pre–post study, 169 medical students’ development of physiology knowledge and their confidence related to this knowledge were assessed via paper-pencil questionnaires before and after attending physiology seminars for one semester. Data from a longitudinal sample of n = 97 students were analyzed using mean comparisons, regression analyses, and latent class analyses (LCAs). In addition, four types of item responses were formed based on confidence and correctness in the knowledge test. Results: We found a significant and large increase in the students’ physiology knowledge, with task-related confidence being the strongest predictor (apart from learning motivation). Moreover, a significantly higher level of confidence at t2 was confirmed, with the level of prior confidence being a strong predictor (apart from knowledge at t2). Furthermore, based on the students’ development of knowledge and confidence levels between measurement points, three empirically distinct groups were distinguished: knowledge gainers, confidence gainers, and overall gainers. The students whose confidence in incorrect knowledge increased constituted one particularly striking group. Therefore, the training of both knowledge and the ability to critically reflect on one’s knowledge and skills as well as an assessment of their development in education is required, especially in professions such as medicine, where knowledge-based decisions made with confidence are of vital importance.
... clinical data) could be digitalized and integrated with these data to further support growth. These data are important because self-assessment without this information often leads to erroneous conclusions (Kruger and Dunning 1999;Eva et al. 2004;Regehr 2005, 2008;Regehr and Eva 2006;Mann et al. 2011). Data are required to identify weaknesses, destabilize selfconceptualizations, flatten hierarchies, identify gaps between perceptions and reality, and alter developmental timelines. ...
Article
Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO’s conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.
... Given that humans are notoriously bad at self-assessment, maintaining appropriate levels of confidence can be challenging if left entirely to the individual to resolve. [21][22][23]51,52 Rather than measuring confidence as an independent variable or construct, we can consider it as being linked to (but not as a surrogate for) competence. The interaction of confidence and competence in HPE has long been considered, [53][54][55][56][57][58] and individuals are viewed to be problematic when confidence and competence are decoupled. ...
Article
Context There have been significant advances in competency-based medical education (CBME) within health professions education. While most of the efforts have focused on competency, less attention has been paid to the role of confidence as a factor in preparing for practice. This paper seeks to address this deficit by exploring the role of confidence and the calibration of confidence with regard to competence. Methods This paper presents a conceptual review of confidence and the calibration of confidence in different medical education contexts. Building from an initial literature review, the authors engaged in iterative discussions exploring divergent and convergent perspectives, which were then supplemented with targeted literature reviews. Finally, a stakeholder consultation was conducted to situate and validate the provisional findings. Results A series of axioms were developed to guide perceptions and responses to different states of confidence in health professionals: 1) confidence can shape how we act and is optimized when it closely corresponds to reality; 2) self-confidence is task-specific, but also inextricably influenced by the individual self-conceptualization, the surrounding system, and society; 3) confidence is shaped by many external factors and the context of the situation; 4) confidence must be considered in conjunction with competence; and 5) the confidence-competence ratio (CCR) changes over time. It is important to track learners’ CCRs and work with them to maintain balance. Conclusion Confidence is expressed in different ways and is shaped by a variety of modifiers. While CBME primarily focuses on competency, proportional confidence is an integral component in ensuring safe and professional practice. As such, it is important to consider both confidence and competence, as well as their relationship in CBME. The CCR can serve as a key construct in developing mindful and capable health professionals. Future research should evaluate strategies for assessing CCR, identify best practices for teaching confidence and guiding self-calibration of CCR, and explore the role of CCR in continuing professional development for individuals and teams.
... Moreover, we decided to limit the number of questions asked to increase the number of participants. As self-assessments can be an unreliable proxy of actual knowledge (29), some of our survey questions may be prone to self-evaluation bias. In addition, data collection took place mostly at the beginning of the pandemic. ...
Article
Full-text available
Background: The emerging field of artificial intelligence (AI) will probably affect the practice for the next generation of doctors. However, the students' views on AI have not been largely investigated. Methods: An anonymous electronic survey on AI was designed for medical and dental students to explore: (1) sources of information about AI, (2) AI applications and concerns, (3) AI status as a topic in medicine, and (4) students' feelings and attitudes. The questionnaire was advertised on social media platforms in 2020. Security measures were employed to prevent fraudulent responses. Mann-Whitney U-test was employed for all comparisons. A sensitivity analysis was also performed by binarizing responses to express disagreement and agreement using the Chi-squared test. Results: Three thousand one hundred thirty-three respondents from 63 countries from all continents were included. Most respondents reported having at least a moderate understanding of the technologies underpinning AI and of their current application, with higher agreement associated with being male (p < 0.0001), tech-savvy (p < 0.0001), pre-clinical student (p < 0.006), and from a developed country (p < 0.04). Students perceive AI as a partner rather than a competitor (72.2%) with a higher agreement for medical students (p = 0.002). The belief that AI will revolutionize medicine and dentistry (83.9%) with greater agreement for students from a developed country (p = 0.0004) was noted. Most students agree that the AI developments will make medicine and dentistry more exciting (69.9%), that AI shall be part of the medical training (85.6%) and they are eager to incorporate AI in their future practice (99%). Conclusion: Currently, AI is a hot topic in medicine and dentistry. Students have a basic understanding of AI principles, a positive attitude toward AI and would like to have it incorporated into their training.
... Given the significant global changes and move to remote learning, how these results generalize and how providers' preferences have already changed is unknown. Finally, our participants' preferences should be viewed in the context of the literature on the limitations of self-assessment and judgments of learning [14]. As such, these findings should guide future studies rather than be viewed as guidance on optimizing CME delivery. ...
Article
Study objective We investigate the continuing medical education delivery preferences among emergency medicine providers, both physicians and advanced practice providers (APPs), within a large, national emergency medicine group. Methods A survey was sent via email to all emergency medicine health care providers in the practice group, including questions evaluating both delivery method and topic preference for continuing education. The study was sent to providers whom the group employed from October 2019 through January 2020. Results A total of 2038 providers, 1373 physicians, and 665 APPs completed the survey - a completion rate of 68.7%. In general, practitioners expressed willingness to learn across various platforms, with a strong overall preference towards online and on-demand options, including video, podcast, and written materials. Universally, a larger percentage of APPs identified a desire for more continuing education across all topics than physicians. Conclusion Education preferences among emergency medicine providers vary with a strong trend toward online and on-demand content. Understanding the delivery and topic preferences for providers is important for the optimal creation of continuing education content.
Article
Background Many medical schools offer a culminating internship readiness experience. Curricula focus on particular knowledge and skills critical to internship, such as answering urgent nursing pages. Studies have shown student performance improvement with mock paging education programs, but the role of feedback versus self-regulated practice has not been studied.Design and Methods The interprofessional mock paging program included 156 medical students enrolled in a 4th-year internship readiness course and 44 master’s level direct entry nursing students. Medical students were randomized to receive verbal feedback immediately after each of the three phone calls (intervention group) or delayed written feedback (control group) after the third phone call only. Specialty-specific case scenarios were developed and a single checklist for all scenarios was developed using the communication tool ISBAR. Medical students and nursing students had separate training sessions before the pages commenced. The nursing students administered the phone calls and evaluated the medical students by ISBAR checklist. An interrater reliability measure was obtained with physician observation of a selection of phone calls.ResultsAfter adjusting for the case effects (different case scenarios for different specialties), students showed no statistically significant differences on checklist scores for case 1 (first case, F = 1.491, df = 1, p = .224), but did show statistically significant differences on checklist scores for case 3 (final case, F = 12.238, df = 1, p = .001). Strong interrater reliability was found between the faculty physician and observed nursing students (ICC = .89).Conclusions Immediate feedback significantly improves student checklist scores with a mock paging program. This finding suggests that coaching with feedback may have advantages above self-regulated learning.
Article
Full-text available
Introduction Citation counts of articles have been used to measure scientific outcomes and assess suitability for grant applications. However, citation counts are not without limitations. With the rise of social media, altmetric scores may provide an alternative assessment tool. Objectives The aims of the study were to assess the characteristics of highly cited articles in medical professionalism and their altmetric scores. Methods The Web of Science was searched for top-cited articles in medical professionalism, and the characteristics of each article were identified. The altmetric database was searched to identify report for each identified article. A model to assess the relationship between the number of citations and each of the key characteristics as well as altmetric scores was developed. Results No correlations were found between the number of citations and number of years since publication (p=0.192), number of institutes (p=0.081), number of authors (p=0.270), females in authorship (p=0.150) or number of grants (p=0.384). The altmetric scores varied from 0 to 155, total=806, median=5.0, (IQR=20). Twitter (54%) and Mendeley (62%) were the most popular altmetric resources. No correlation was found between the number of citations and the altmetric scores (p=0.661). However, a correlation was found for articles published in 2007 and after (n=17, p=0.023). To further assess these variables, a model was developed using multivariate analysis; did not show significant differences across subgroups. The topics covered were learning and teaching professionalism, curriculum issues, professional and unprofessional behaviour. Conclusions Altmetric scores of articles were significantly correlated with citations counts for articles published in 2007 and after. Highly cited articles were produced mainly by the USA, Canada and the UK. The study reflects the emerging role of social media in research dissemination. Future studies should investigate the specific features of highly cited articles and factors reinforcing distribution of research data among scholars and non-scholars.
Article
Introduction Selecting medical students for residency is a competitive process, with a narrow range of scores separating middle-ranked applicants. Self-assessment is a fundamental skill for any competent physician with a demonstrated correlation to diagnostic ability, examination scores, and technical skills, but has yet to be investigated in residency selection. The objective of this study was to investigate the relationship between self-assessment and interview performance as a potential adjunct to discriminate between applicants. Methods At the University of Ottawa in 2020, 55 applicants completed a 9-station interview circuit assessing different characteristics or skills important for a career in general surgery, followed by a self-assessment questionnaire evaluating their perceived performance at each station. Pearson's correlation was used to determine the relationship between self-assessment scores (SASs) and interviewer scores (ISs). Results There was a negative correlation between SASs and ISs for all interview stations. High performers underestimated their interview performance, and low performers overestimated their performance. Seven of the nine stations reached statistical significance (r = 0.60-0.73, P < 0.001). There was significant variability in the SAS of middle-ranked applicants, with a range three times greater than the range of ISs and demonstrating distinct self-assessment skills in candidates with very similar scores. Conclusions Although we strive to select applicants who will succeed in residency to become competent physicians, self-assessment skills may be a useful adjunct during the interview process to assist in discriminating between applicants with similar scores.
Article
Introduction The objective of this study was to compare faculty member evaluations to student self-evaluations in a clinical endodontic course in the dental school at King Saud University and to evaluate the reliability of the students’ self-assessment scores after using a rubric with well-defined criteria. Materials and methods Evaluated and self-evaluated endodontic cases that were clinically treated by fourth-year undergraduate dental students at the College of Dentistry, Girls University Campus (GUC), at King Saud University over two years (2017-2018) were included. Cases included anterior teeth, premolars, and molars. The evaluation form was divided into six sections with well-defined criteria to cover all aspects of nonsurgical root canal treatment with a maximum grade of 10 points can be scored for each student per case. The students evaluated themselves for each section, and then were evaluated by two faculty members. Student and faculty assessment agreement and the reliability of the students’ self-assessment scores were measured. A P-value ≤ 0.05 was considered significant. Results A total of 363 cases were included: 26.7% anterior teeth, 38.84% premolars, and 34.43% molars. The students evaluated themselves with higher grades compared to the evaluators’ grading in all steps and in the overall grading in all teeth types. The students’ self-assessment scores showed good and moderate reliability in all steps and in the overall grading. Conclusion The students tend to overrate their performance and their assessments have moderate to good reliability, which reflects the reliability of the rubric used as an accurate measurement tool that helps the evaluator and the student objectively assess their performance.
Article
Purpose: Compare time (speed) and product quality goals in a surgical procedural task. Methods: Secondary school students participating in a medical simulation-based training activity participated in a randomized experiment. Each participant completed eight repetitions of a blood vessel ligation. Once, between repetitions four and five, each participant received a randomly-assigned speed goal or quality goal. Outcomes included time and leak-free ligatures. Results: 80 students participated. The speed-goal group performed 18% faster on the final repetition than the quality-goal group, with adjusted fold change (FC) 0.82 (95% confidence interval [CI], 0.71, 0.94; p = 0.01). Conversely, the speed-goal group had fewer high-quality (leak-free) ligatures (odds ratio [OR] 0.36 [95% CI, 0.22, 0.58; p < 0.001]). For the quality-goal group, leaky ligatures took longer post-intervention than leak-free ligatures (FC 1.09 [95% CI, 1.02, 1.17; p = 0.01]), whereas average times for leaky and leak-free ligatures were similar for the speed-goal group (FC 0.97 [95% CI, 0.91, 1.04; p = 0.38]). For a given performance time, the speed-goal group had more leaks post-intervention than the quality-goal group (OR 3.35 [95% CI, 1.58, 7.10; p = 0.002]). Conclusions: Speed and quality goals promote different learning processes and outcomes among novices. Use of both speed and quality goals may facilitate more effective and efficient learning.
Article
Full-text available
Curriculum reform has been a central theme for many dental schools through the revised predoctoral standards; and many curricular changes involve interdisciplinary teaching, integration, and problem-based and critical thinking exercises. However, it is also necessary to align our assessments to match our curriculum objectives in order to reach our education outcomes. To be able to stay update in dental education, we have to ensure that our students reach at least minimal levels of competence on many different fronts. Knowledge base, attitudes, values, and skills are all a part of what we are asked to assess on a daily basis. Choosing the right type of assessment in each area is critical to our students’ success. Various evaluation methods such as case presentation, direct observation, daily clinical grades, feedbacks, peer evaluation and clinically structured examinations for the assessment of skills, competencies and performances are being discussed in this review study. KEYWORDS Dental education, assessment, skill, competence
Article
Full-text available
Faculty development: Roles and competences for educators of undergraduate dental students Development of faculties is inevitable because of changing in learning and teaching strategies and improvements of dental materials and treatments. Dental educators are important people who contribute to the development of every aspect of dental education. They assume lots of roles and responsibilities including curriculum development, teaching and education. This article presents common roles for educators of undergraduate dental students and the competences needed to be effective educators; and the basic procedures relating curriculum development. Being a lecturer, profesional expert or faculty member do not automatically ensure having good teaching expertise. Education has no chance to develop if educators only believe in methods of their old teachers. It is necessary to be aware of the competencies of dental educators. Educators need to have macro level competences such as educational programme and curriculum matters togehter with educational principles and strategies, student issues and assessment competences that relate to the micro‐level of education.
Article
Purpose: Objective of this study was to determine if perceptual abilities test (PAT), quantitative reasoning (QR), and academic average (AA) scores predict performance in a rudimentary dental anatomy laboratory exercise. Methods: First-year dental students (n = 88) completed two identical wax carving exercises during the first and last weeks of the dental anatomy course. After being given detailed instructions and a wax block, students carved a cube and a semilunar shape using the wax subtraction technique. Using a rubric, students self-assessed their work as unsatisfactory, satisfactory, or ideal in the following areas: finish, sharpness, symmetry, and accuracy. Two calibrated and blinded faculty graded the exercise using the same rubric as the students. Cohen's kappa coefficient assessed faculty interrater reliability and student-faculty reliability. Changes in scores between the two exercises were evaluated using Wilcoxon signed rank and paired t-tests. Spearman's correlation coefficient and repeated measures analysis assessed the association between shape carving scores and DAT, QR, and AA scores. Results: Faculty interrater reliability ranged from 0.2554 to 0.4186 for the assessed criteria over the two exercises. Student-faculty rater reliability ranged from 0.1299 to 0.4461. Only PAT scores correlated significantly to higher scores for the four grading criteria for both exercises. Student self-assessment improved with the second exercise. Conclusions: This rudimentary exercise was useful to identify students early in their dental education who may require more guidance in other manual dexterity courses. Faculties require additional training and calibration. Self-assessment is a life-long process of critical thinking that students are just beginning to learn.
Article
Self-assessment, or students’ evaluation of their own learning or performance in academic tasks, is a self-regulatory process that is intertwined with learners’ cognitive, behavioral, and motivational processes. Among the motivational beliefs that have been studied in relation to self-assessment is self-efficacy, which refers to beliefs about one's capability to learn or perform a specified task successfully. The current study describes the development and validation of the Self-Efficacy for Self-Assessment in Argumentative Writing (SEESA-AW) scale to measure college students’ self-efficacy beliefs for self-assessment in the domain of argumentative writing with two samples of undergraduate college students (N=335 and N=662). Exploratory and confirmatory factor analyses revealed three factors (grammar & mechanics, cohesiveness/flow, the presentation of arguments) that explained 65.4% of the variance. Evidence is also provided for convergent validity of the scale with regard to other writing self-efficacy scales. Educational implications are discussed.
Article
The purpose of this paper is to study the effectiveness of oral presentation as an assessment tool in a Finance subject. Assessment data collected from a postgraduate Finance subject in an Australian university over a period of five years from 2005 to 2009 was analysed statistically to determine the relation between students’ performance in oral presentation and other forms of assessments. The sample consists of assessment records of 412 students and 98 group presentations. From the study of correlations between oral presentations and other assessments, it is concluded that students perform better in written assessments compared to oral assessment. The study of effect of gender on students’ performance leads to the conclusion that female students perform better than male students in all forms of assessments except oral presentations where male students perform better although difference between males and females in oral presentation is not very large. The study of students’ performance based on their nationality leads to the conclusion that domestic students perform better than international students in all forms of assessments. Based on the study of student’ performance in oral presentation, it is found that students did well in the development of content of presentations, quality of their analysis, group coordination and organisation of presentation. There is however a general tendency to treat group work as a sum of parts instead of treating the group work as a single task. This study is limited by the fact that effectiveness of oral presentation is studied in only one Finance subject. This study makes an original contribution to the literature as the effectiveness of oral assessment in Finance subject is being studied for the first time. The conclusions arrived in this paper have many implications for policies and practice of learning and teaching in Finance.
Chapter
The purpose of assessment in mastery learning is to promote learner improvement. Assessment is a key and indispensable feature of the mastery learning bundle. Mastery learning approaches are linked to the ability to make accurate decisions. Accurate decisions depend on reliable data that have been collected using instruments and methods linked to curriculum learning objectives. This chapter has six sections that reveal the ways that instruction and assessment are inseparable in mastery learning: (a) curriculum and instruction, (b) validity argument, (c) assessment context, (d) assessment measures, (e) data, and (f) decisions. Health professions educators must collect validity evidence regularly to support the validity argument that they are making accurate decisions about learners at each step of assessment design and implementation.
Article
Full-text available
Regulation has traditionally been explored in terms of its negative burden on financial performance. We posit a true assessment should consider both its burden and value, as well as financial and wider performance criteria. This hypothesis is explored in the context of micro-firms using original survey data from the English tourism industry, using multivariate regression techniques and qualitative comments. Perceived-Burden demonstrates a negative association with financial performance, while Perceived-Value demonstrates a positive association with goal-satisfaction. Resulting policy recommendations focus on accompanying future regulatory changes with appropriate positive information so managers can move beyond burden to recognize the inherent value, and suggesting the need for further research in this area.
Article
Background Accurate self-assessment of knowledge and technical skills is key to self-directed education required in surgical training. We aimed to investigate the presence and magnitude of cognitive bias in self-assessment among a cohort of surgical interns. Methods First-year general surgery residents self-assessed performance on a battery of technical skill tasks (knot tying, suturing, vascular anastomosis, Fundamentals of Laparoscopic Skills peg transfer and intracorporeal suturing) at the beginning of residency. Each self-assessment was compared to actual performance. Bias and deviation were defined as arithmetic and absolute difference between actual and estimated scores. Spearman correlation assessed covariation between actual and estimated scores. Improvement in participant performance was analyzed after an end-of-year assessment. Results Participants (N = 34) completed assessments from 2017 to 2019. Actual and self-assessment scores were positively correlated (0.55, P < .001). Residents generally underestimated performance (bias -4.7 + 8.1). Participants who performed above cohort average tended to assess themselves more negatively (bias -7.3 vs -2.3) and had a larger discrepancy between self and actual scores than below average performers (deviation index 9.7 + 8.2 vs 3.8 + 3.1, P < .05). End-of-year total scores improved in 31 (91.2%) participants by an average of 11 points (90 possible). Least accurate residents in initial self-assessments (deviation indices >75th percentile) improved less than more accurate residents (median 5 vs 16 points, P < .05). All residents with a deviation index >75 percentile underestimated their performance. Conclusion Cognitive bias in technical surgical skills is apparent in first-year surgical residents, particularly in those who are higher performers. Inaccuracy in self-assessment may influence improvement and should be addressed in surgical training.
Article
Study objectives: The referral burden on healthcare systems for routine sleep disorders could be alleviated by educating primary care providers (PCPs) to diagnose and manage patients with sleep health issues. This requires effective professional education strategies and resources. This scoping review examined the literature on existing approaches to educate PCPs in sleep health management. Methods: A comprehensive literature search was conducted across eight databases to identify citations describing the education of PCPs in diagnosing and managing sleep disorders, specifically insomnia and sleep apnea. A conceptual framework, developed from the knowledge-to-action cycle, was used to analyze citations from a knowledge translation perspective. Results: Searches identified 616 unique citations and after selection criteria were applied, 22 reports were included. Reports spanning 38 years were analyzed using components of the knowledge-to-action cycle to understand how educational interventions were designed, developed, implemented, and evaluated. Interventions involved didactic (32%), active (18%) and blended (41%) approaches, using face-to-face (27%), technology-mediated (45%) and multimodal (5%) delivery. Educational effectiveness was assessed in 73% of reports, most commonly using a pre/post questionnaire (41%). Conclusions: While this scoping review has utility in describing existing educational interventions to upskill PCPs to diagnose and manage sleep disorders, the findings suggest that interventions are often developed without explicitly considering the evidence of best educational practice. Future interventional designs may achieve greater sustained effectiveness by considering characteristics of the target audience, the pedagogical approaches best suited to its needs, and any environmental drivers and barriers that might impede the translation of evidence into practice.
Article
The quality of experiential education for pharmacists‐in‐training is heavily impacted by the habits of their preceptors. Assessment of preceptor capabilities and evidence of development has been limited due to the absence of a tool to meaningfully facilitate preceptor assessment in a robust and progressive fashion. The primary objective was to determine the change in perceived precepting behavior and abilities of residents and preceptors enrolled in a teaching and learning curriculum (TLC) using a self‐directed assessment seeking (SDAS) approach via the Habits of Preceptors Rubric (HOP‐R). Pharmacists enrolled in the 2018 to 2019 cohort of the Clinician Educators Program (CEP) TLC were invited to assess their precepting habits using the HOP‐R after attending the first seminar and again toward the end of the longitudinal 12‐month program using a SDAS approach. Using online surveys, participants rated their precepting capabilities as developing, proficient, accomplished, or master level for each of the 11 preceptor habits within the HOP‐R. The change in their self‐reported ratings was evaluated. The response rate for participants completing both the initial and follow‐up surveys was 56% (52/93). When comparing matched responses, the difference in self‐reported habit level ratings was statistically significant. The overwhelming majority of participants, 94%, reported advancement in one or more habits, with 60% reporting advancement in six or more habits while enrolled in the CEP. Preceptor and resident pharmacists reported advancement of precepting capabilities within the conceptual framework of the HOP‐R while enrolled in a TLC. The SDAS approach yielded participant comments showcasing the perceived value of completing the criterion‐referenced HOP‐R assessment to foster self‐awareness and self‐efficacy related to precepting and preceptor development. The HOP‐R can be utilized within TLC to allow participants to assess precepting habits and show growth and development of their capabilities over time both individually and in aggregate.
Article
Full-text available
To compare how well graduates of a self-directed, problem-based undergraduate curriculum (at McMaster University [MU], Hamilton, Ont.) and those of a traditional curriculum (at the University of Toronto [UT]) who go on to primary care careers keep up to date with current clinical practice guidelines. Analytic survey. Management of hypertension was chosen as an appropriate topic to assess guideline adherence. An updated version of a previously validated questionnaire was mailed to the participants for self-completion. Private primary care practices in southern Ontario. A random sample of 48 MU graduates and 48 UT graduates, stratified for year of graduation (1974 to 1985) and sex, who were in family or general practice in Ontario; 87% of the eligible subjects in each group responded. Overall and component-specific scores; analysis was blind to study group. The overall mean scores were 56 (68%) of a possible 82 for the MU graduates and 51 (62%) for the UT graduates (difference between the means 5, 95% confidence interval 1.9 to 8.2; p < 0.01). Multivariate linear regression analysis of seven factors that might affect questionnaire scores revealed that only the medical school was statistically significant (p < 0.01). The MU graduates had significantly higher mean scores than the UT graduates for two components of the questionnaire: knowledge of recommended blood pressures for treatment (p < 0.05) and successful approaches to enhance compliance (p < 0.005). The trends were similar for the other components but but were not significant. The graduates of a problem-based, self-directed undergraduate curriculum are more up to date in knowledge of the management of hypertension than graduates of a traditional curriculum.
Article
Full-text available
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
Article
Full-text available
The competent physician pursues lifelong learning through the recognition of deficiencies and the formulation of appropriate learning goals. Despite the accepted theoretical value of self-assessment, studies have consistently shown that the accuracy of self-assessment is poor. This paper examines the methodological issues that plague the measurement of self-assessment ability and presents several strategies that address these methodological problems within the current paradigm. In addition, the article proposes an alternative conceptualization of self-assessment and describes its associated methods. The conclusions of prior research in this domain must be re-examined in light of the common pitfalls encountered in the design of the studies and the analyses of the data. Future efforts to elucidate self-assessment phenomena need to consider the implications of this review.
Article
People feel they understand complex phenomena with far greater precision, coherence, and depth than they really do; they are subject to an illusion-an illusion of explanatory depth. The illusion is far stronger for explanatory knowledge than many other kinds of knowledge, such as that for facts, procedures or narratives. The illusion for explanatory knowledge is most robust where the environment supports real-time explanations with visible mechanisms. We demonstrate the illusion of depth with explanatory knowledge in Studies 1-6. Then we show differences in overconfidence about knowledge across different knowledge domains in Studies 7-10. Finally, we explore the mechanisms behind the initial confidence and behind overconfidence in Studies 11 and 12. Implications for the roles of intuitive theories in models of concepts and cognition are discussed.
Article
Although expected of all health professionals, self-assessment skills are seldom addressed directly in training. A previous review by the author identified curricular criteria associated with improved accuracy and validity in self-assessments of knowledge and performance in curriculum studies published between January 1970 and February 1990. The present review analyzed 11 studies that meet those criteria. Eight studies were of implementations of self-assessment components within training programs in the health professions, and three involved other training environments. Most described initial disorientation or opposition on the part of learners, attributed to unfamiliar roles and to learners' distrust. The curricula that successfully negotiated the transition to self-assessment norms reported noncognitive benefits such as improvements in morale, motivation, and communications among learners and faculty. Reported cognitive benefits included improvements in knowledge, performance, and self-analysis of performance. The constellation of effects suggests that effective self-assessment programs may promote more mature, collegial, and productive learning environments, particularly suited to the training of health professionals. Most curricula fostering effective self-assessment did not require extraordinary resources, and none jeopardized traditional standards. No evidence was found to support or challenge the expectation that self-assessment training would transfer to later work settings.
Article
Valid self-assessment is fundamental to continuing professional competence but is seldom explicitly taught in health professions training. This review analyzed 18 scholarly articles published between January 1970 and February 1990 (14 articles regarding health professions trainees, and four concerning college students or graduate trainees) in which it was possible to compare performance as self-assessed by trainees with performance as assessed by experts or objective tests. The validity of self-assessed performance was found to be low to moderate and did not improve with time in conventional health professions training programs. Self-assessed performance seemed closely related to generalized self-attributions and was minimally influenced by external feedback in the form of test scores, grades, or faculty assessments. In five programs emphasizing explicit self-assessment goals and training strategies, moderate-to-high validity outcomes or improvements over time were demonstrated. Much of what passes for self-assessment in training seems the exercise of an underdeveloped skill, but effective training to improve validity and accuracy is available and feasible.
Article
Medical students in a tutorial program rated themselves in Year 1 and again in Year 2 on Knowledge, Reasoning, Communication and Interpersonal Skills. Student and tutor ratings were similar in Year 1 but different in Year 2. Both self and tutor ratings increased from Year 1 to Year 2 but self-ratings were independent of achievement while tutor ratings were related to achievement. Possible reasons for these results were discussed with reference to the learning goals of the tutorials. © 1989 Informa UK Ltd All rights reserved: reproduction in whole or part not permitted.
The purpose of this paper was to determine if the decline in performance with time since completion of training on the 1980 ABIM Recertification Examination can be explained by a difference in performance on items testing different types of knowledge. Results showed that candidates further out of training performed less well on items testing new or changing knowledge, while performance on items testing stable knowledge was relatively constant across age groups.
Article
Although self-evaluation is crucial in the practice of medicine, few educators have formally introduced self-assessment into the undergraduate medical curriculum. However, students in the baccalaureate-M.D. degree program at the University of Missouri, Kansas City, must complete a self-evaluation at the close of every medical school course and rotation during the last four years of a six-year curriculum. In this paper, the authors examine the self-ratings of 211 of these students as they progressed through the program in order to discover trends in and correlates of the self-assessments. Although the students' self-evaluations and faculty members' ratings of these students' performances rose year by year, the relationship between the students' and the faculty's ratings decreased through time. Yet, results suggest that self-evaluation has educational merit as a measure of noncognitive abilities associated with clinical performance and as a stimulus to further learning and professional development.
Article
To determine whether continuing medical education affects the quality of clinical care, we randomly allocated 16 Ontario family physicians to receive or not receive continuing-education packages covering clinical problems commonly confronted in general practice. Over 4500 episodes of care, provided before and after study physicians received continuing education, were compared with preset clinical criteria and classified according to quality. Although objective tests confirmed that the study physicians learned from the packages, there was little effect on the overall quality of care. When the topics were of relatively great interest to the physicians, the control group (who did not receive the packages) showed as much improvement as did the study group. When the topics were not preferred, however, the documented quality of care provided by study physicians rose (P less than 0.05) and differed from that provided by control physicians (P = 0.01). Finally, there was no spillover effect on clinical problems not directly covered by the program. In view of the trend toward mandatory continuing education and the resources expended, it is time to reconsider whether it works.
Article
As physicians, whether serving individual patients or populations, we always have sought to base our decisions and actions on the best possible evidence. The ascendancy of the randomized trial heralded a fundamental shift in the way that we establish the clinical bases for diagnosis, prognosis, and therapeutics. The ability to track down, critically appraise (for its validity and usefulness), and incorporate this rapidly growing body of evidence into one's clinical practice has been named 'evidence-based medicine' (EBM).
Article
The goal of the study was to examine critically the use of taxonomic level as a criterion for determining the utility of a test item. If the distinction is useful, several hypotheses follow: There should be reasonable agreement among raters about taxonomic level; the taxonomic level should be independent of the level of the rater; in contrasting students from PEL and conventional curricula, since PEL students consistently report that they spend more time on conceptualizing and less on memorizing, there should be differential performance on factual recall and higher order items; higher order items should have greater predictive validity (addressed in other studies). Test items from the 1993 and 1994 Medical Council of Canada multiple choice examinations were reviewed by two recent graduates and two faculty. Although there was moderate agreement among raters, faculty were significantly more likely to rate items as Factual Recall. Performance of McMaster students and other Canadian graduates were then compared at an item level. No systematic differences between the two cohorts related to taxonomic level were identified. In conclusion, while there is moderate agreement about the taxonomic level of a test item, this appears to be related to educational level, and has little heuristic value as an indicator of performance.
Article
The progress test (or Quarterly Profile Examination), invented concurrently by the University of Missouri-Kansas City School of Medicine and the University of Limburg, is used to provide useful summative and formative judgments about students' knowledge without distorting learning. All students in all classes sit the same examination at regular intervals through the year, and their individual progress is noted. This paper reports on four years' experience with a progress test, the Personal Progress Index (PPI), at the McMaster University Faculty of Health Sciences. The PPI, a 180-item multiple-choice test with items drawn from all disciplines of medicine, is administered to medical students in all three classes three times per year. Individual feedback is provided, and accumulated student performance is determined for summative purposes. This paper examines extensive evidence of reliability, validity, and effect on student learning, using samples from the entering classes of 1992-1995 (a total of 400 students). Reliabilities across test administrations (within classes) ranged from .46 to .63. The PPI demonstrated strong construct validity, with highly significant statistical tests of differences between classes and changes within classes on successive administrations. The predictive validity of the PPI, i.e., whether it could predict performance on the licensing examination of the Medical Council of Canada, increased monotonically from a correlation of .12 for the first test administration one month into medical school to a high of about .60 for the cumulative score across all administrations three months prior to the examination. The PPI seems to be performing as intended, with students showing growth in performance across the three years of the MD program. Additional benefits are that many more students now self-refer for remediation (based on low PPI scores) and that the consistent relative performances of individual students across test administrations allow for the identification of students who have severe and persistent problems.
Article
Although studies have examined medical students' ability to self-assess their performance, there are few longitudinal studies that document the stability of self-assessment accuracy over time. This study compares actual and estimated examination performance for three classes during their first 3 years of medical school. Students assessed their performance on classroom examinations and objective structured clinical examination (OSCE) stations. Each self-assessment was then contrasted with their actual performance using idiographic (within-subject) methods to define three measures of self-assessment accuracy: bias (arithmetic differences of actual and estimated scores), deviation (absolute differences of actual and estimated scores), and covariation (correlation of actual and estimated scores). These measures were computed for four intervals over the course of 3 years. Multivariate analyses of variance and correlational analyses were used to evaluate the stability of these measures. Self-assessment accuracy measures were relatively stable over the first 2 years of medical school with a decease occurring in the third year. However, the correlational analyses indicated that the stability of self-assessment accuracy was comparable to the stability of actual performance over this same period. The apparent decline in accuracy in the third year may reflect the transition from familiar classroom-based examinations to the substantially different clinical examination tasks of the third year OSCE. However, the stability of self-assessment accuracy compares favorably with the stability of actual performance over this period. These results suggest that self-assessment accuracy is a relatively stable individual characteristic that may be influenced by task familiarity.
The ends, and beginnings, or science in the head: coming to grasp the causal structure of the world
  • F Keil