No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
How Can I Know What I Don't Know? Poor Self Assessment in a Well-Defined Domain
Abstract
As the rapidity with which medical knowledge is generated and disseminated becomes amplified, an increasing emphasis has been placed on the need for physicians to develop the skills necessary for life-long learning. One such skill is the ability to evaluate one's own deficiencies. A ubiquitous finding in the study of self-assessment, however, is that self-ratings are poorly correlated with other performance measures. Still, many educators view the ability to recognize and communicate one's deficiencies as an important component of adult learning. As a result, two studies have been performed in an attempt to improve upon this status quo. First, we tried to re-define the limits within which self-assessments should be used, using Rosenblit and Keil's argument that calibration between perceived and actual performance will be better within taxonomies that are regularly tested (e.g., factual knowledge) compared to those that are not (e.g., conceptual knowledge). Second, we tried to norm reference individuals based on both the performance of their colleagues and their own historical performance on McMaster's Personal Progress Inventory (a multiple choice question test of medical knowledge). While it appears that students are able to (a) make macro-level self-assessments (i.e., to recognize that third year students typically outperform first year students), and (b) judge their performance relatively accurately after the fact, students were unable to predict the percentage of questions they would answer correctly with a testing procedure in which they have had a substantial amount of feedback. Previous test score was a much better predictor of current test performance than were individuals' expectations.
... Furthermore, self-assessment is an important method in education, as it can be an effective learning tool. 8,9 Burrows 7 suggests that SA should be a collaborative process, with teachers and students developing reflective skills through mutual feedback. ...
... 4,5 Furthermore, effective feedback is also essential for effective self-assessment in improving student educational outcomes. 7,8 The SimEx dental education and evaluation system can provide students with accurate 3D real-time feedback, digital guidance and simulation, as well as objective computer-based assessment, facilitating student self-study and selfassessment and improving clinical skills. 10 These could be the reasons why for Q1 and Q4, all groups scored 4 or higher. ...
Background/purpose:
The SimEx is dental training system that applies new technology in a computerized dental simulator. The purpose of this study was to understand the usability satisfaction of the SimEx dental education and evaluation system by dental students and dentists at Tohoku University.
Materials and methods:
In this study, the Tohoku University IRB execution number was 2020-3-33. The number of subjects accepted was 59 at Tohoku University and divided into 4 groups based on years of clinical experience (Group A: 0 years; Group B: 1-2 years; Group C: 2-5 years; Group D: at least 5 years), and a total of 58 usability questionnaires were collected. Subjects completed the SimEx Usability Satisfaction Questionnaire after operating the SimEx (EPED Inc., Kaohsiung, Taiwan) course, which contained 16 questions.
Results:
Among the 58 questionnaires collected by Tohoku University, there were 19 undergraduate students (4th∼6th grade), 12 post-graduate students, 14 residents, and 13 dentists. Significant differences between Group A and Group B, and between Group A and Group D were found (P < 0.05). The same results were obtained for the "experience satisfaction index". In the items where significant differences were found, longer clinical experience tended to result in lower scores.
Conclusion:
From these results, we can conclude that the SimEx education and evaluation system facilitates students' self-learning, and this system is very useful for continued study and clinical skill training for dentists, especially for students and junior dentists with high usability satisfaction.
... Furthermore, since the subject of the study is the perception of the owner-manager, who is generally the sole worker in the firm (or on average, has just one further employee), it is not possible to survey multiple respondents per firm. Face validity and subjective assessment validity (Hair, 1998;Hair et al., 2010) were satisfied through the use of existing methods of measurement where possible, such as Stokols et al. (2001) and Eva et al. (2004) for Perceived-Knowledge; and the pilot study (which ensured the understanding of questions and answers matched our expectations, Rong and Wilkinson, 2011). In addition, the survey was structured specifically to reduce the risk of respondents linking different concepts. ...
... Cronbach's alpha = 0.79. This variable is based upon prior literature including Stokols et al. (2001) and Eva et al. (2004), who both employ similar measures to test self-assessments of knowledge. Perceived-knowledge is used here, rather than a test of actual-knowledge because opinions of value, burden, and compliance level are based on what owner-managers think they know, rather than their actual knowledge, which is likely different ...
Business regulation is only successful when firms comply. Compliance is determined by several factors, including knowledge-of/attitude-towards regulation, which we suggest consists of both burden and value. Existing literature generally ignores regulatory value and focuses on larger firms in “high-risk” industries. We explore Perceived Own-Firm and Perceived Competitor Compliance in English accommodation micro-firms. Perceived Own-Firm Compliance exceeds Perceived-Knowledge of regulation, and has the strongest (positive) association with Perceived-Value. Perceived Competitor Compliance is thought to be very low and is (negatively) associated with Perceived-Value and Perceived-Burden. Furthermore, the factors associated with greater Perceived Own-Firm Compliance are associated with lower Perceived Competitor Compliance.
... Sense of Urgency Vs Actual Urgency Students described feeling a sense of urgency to treat the patient quicker, however this did not align with their actual time to treat and decision making patterns.This suggests that participants' perceptions of performance may not align with actual performance. This phenomenon is not new -Eva et al (2004) demonstrate the correlation between self-assessment and actual performance in students is poor[194] -"How can I know what I don't know?". In a practical sense, authentic portrayal of moulage might provide more opportunities for students to bridge the gap between self-awareness and performance; moulage presents an opportunity to deliver feedback from objective ...
... Sense of Urgency Vs Actual Urgency Students described feeling a sense of urgency to treat the patient quicker, however this did not align with their actual time to treat and decision making patterns.This suggests that participants' perceptions of performance may not align with actual performance. This phenomenon is not new -Eva et al (2004) demonstrate the correlation between self-assessment and actual performance in students is poor[194] -"How can I know what I don't know?". In a practical sense, authentic portrayal of moulage might provide more opportunities for students to bridge the gap between self-awareness and performance; moulage presents an opportunity to deliver feedback from objective ...
Moulage in the traditional sense is the art of replicating illnesses and wounds through casting wax moulds. Origins are traced to Ancient Egypt and forbidden practices of 17th century Europe. While traditional moulage is now housed in musea across the world, modern moulage is used to replicate illness and effects in simulation using special effects makeup techniques. Simulation is a well-established technique to prepare health professionals for clinical practice, and is grounded in a strong evidence base. Despite the strong evidence for the use of simulation, the conditions of moulage is an underexplored topic within the context of simulation research, and we know very little regarding how it works, under what conditions and what the effect is on participants of simulation. In order to better understand how and why moulage impacts on participants of simulation, a series of complementary studies were completed. Initially a Systematic Review of authentic moulage in simulation was undertaken to understand the current research on moulage. This provided a useful baseline for the current use and evidence for moulage in simulation. Subsequently, a further study was undertaken to define authentic moulage in simulation via an electronic Delphi consensus method. This study recruited international experts on moulage and resulted in the development of the Moulage Authenticity Rating Scale (MARS) to measure moulage authenticity. Finally, a third study was conducted to explore how the authenticity of moulage effects participant engagement in simulation using a randomized control experiment design. This study utilized the MARS tool developed from the previous study to design moulage that was low-authenticity and high-authenticity, and compared levels of engagement using measures of self-report, eye tracking and interview methods. The results of this work presents previously unrecognized information on how medical students perceive the authenticity of moulage and how it contributes to their performance and engagement in simulation. In summary, I present a number of suggestions as to how simulation users and designers might consider moulage in their everyday practice. This thesis presents a series of philosophical research questions and findings that collectively make an original contribution to the future of moulage in simulation and undergraduate Medical Education using simulation-based curriculum, teaching and learning.
http://hdl.handle.net/1959.13/1412570
... Such contradiction may put forward the idea of leaving the responsibility of assessment to learners. However, such ideas have proven to be too controversial (Deci & Ryan, 2008;Eva et al., 2004). It is also argued that even though PBL is conducive to students' effective learning, it seems challenging to assess learners' performance during the course (Shamsan & Syed, 2009). ...
Problem-based learning (PBL), as one of the trendy learning and teaching approaches, has been investigated in various disciplines. Following the cresting benefit and reputation of this methodology in medicine (Vleuten & Schuwirth, 2019), it soon grabbed scholars' attention in the field of language teaching. The present exploratory mixed-method study was an attempt to identify the potentially impeding factors that prevent language teachers from offering problem-based teaching in their classes despite its reported advantages. To this end, in the qualitative phase, 20 EFL teachers took part in a semi-structured interview. The interviews were transcribed, coded, categorized, and thematized. In the quantitative phase, a 66-item Likert scale questionnaire was developed based on the qualitative phase results and was administered to 200 EFL teachers. The results indicated that factors such as difficulties in designing a PBL problem/scenario, the role of teaching equipment, teachers' unfamiliarity with designing a problem/scenario, limiting PBL to a specific skill, assessment, students/teachers' unfamiliarity with the PBL, students' cross-cultural differences, and limiting PBL to a specific level of language proficiency discourage teachers from using this approach. The implications of the findings are discussed and suggestions are made for future research.
... Several theories, including cognitive load theory, encourage maximum coaching early in the learning process [11,12]. Findings also suggest that learners struggle to self-assess their learning [13] and thus may make poor use of independent practice opportunities. Consistent with this, a recent study found that expert feedback during early deliberate practice supported greater learning of endourologic skills than did feedback provided during a later deliberate practice session [14]. ...
Background
Opportunities to practice procedural skills in the clinical learning environment are decreasing, and faculty time to coach skills is limited, even in simulation-based training. Self-directed learning with hands-on practice early in a procedural skill course might help maximize the benefit of later faculty coaching and clinical experience. However, it may also lead to well-learned errors if learners lack critical guidance. The present study sought to investigate the effects of a hands-on, self-directed “study hall” for central line insertion among first-year residents.
Methods
Learner cohorts before vs. after introduction of the study hall ( n = 49) were compared on their pre- and post-test performance of key procedural behaviors that were comparable across cohorts, with all learners receiving traditional instructor-led training between tests.
Results
Study hall participants spent a median of 116 min in hands-on practice (range 57–175). They scored higher at pre-test (44% vs. 27%, p = .00; Cohen’s d = 0.95) and at post-test (80% vs. 72%, p = .02; Cohen’s d = 0.69). A dose–response relationship was found, such that 2 h of study hall were roughly equivalent to the performance improvement seen with four clinical observations or supervised insertions of central lines.
Conclusions
Self-directed, hands-on “study hall” supported improved procedural skill learning in the context of limited faculty availability. Potential additional benefits make the approach worth further experimentation and evaluation.
... One weakness of all three of these inventories is that they are learning tools rather than formal tests of skill sets. In addition, there is a long-standing discussion across a wide range of disciplines about the accuracy of self-assessments when compared to observed measures of competence or objective tests (Falchikov & Boud, 1989;Wilson, 1999;Kruger & Dunning, 1999;Eva et al., 2004;Dunning et al., 2004;Davis et al., 2006). For example, in a meta-analysis of quantitative self-assessment studies in law, engineering, guidance counseling, behavioral science, psychology and medicine, Falchikov and Boud found correlations between self and external assessments of student performance ranging from .05 to .82, with a mean of .39. ...
This study investigates students' self-perceptions of learning related to their participation in either a 3- or 15-week skill-based graduate coaching course. The results indicate students enrolled in the compressed course saw a significant increase in their coaching skills from pre-test to post-test; students enrolled in the full-term did not see the same statistically significant increase. Further, students in the compressed course had statistically significant higher counseling and challenging skills than the full-term students. These findings are relevant to the debates surrounding the benefits accrued from obtaining an MBA. (original abstract)
... 24 Another advantage of CBL is that it promotes selfdirected learning (SDL), which is an important component of lifelong learning and therefore a vital competency in medical school curricula. 25 Various studies have described the use of CBL in anatomy education through graded oral presentations designed by the first-year medical students, 26 as well as the positive perception and efficacy of CBL in anatomy education compared to other teaching strategies. [27][28][29] One important but commonly overlooked aspect is the role of structured reviews in active learning. ...
Introduction:
Recent changes in undergraduate medical curricula have resulted in time constraints that are particularly challenging, especially when students must learn large blocks of musculoskeletal anatomy content. Consequently, students have resorted to rote memorization to cope, which counteracts our established collaborative self-directed learning model.
Methods:
For a 6-week musculoskeletal anatomy course, two structured case-based review sessions are described, each following the completion of two five-hour lab sessions, two on the upper extremities and two on the lower extremities. These largely self-directed review sessions consisted of 6 students rotating through 7 to 8 stations every 10 minutes where clinical cases with follow-up questions were projected on large screens. The students were expected to work collaboratively to solve the cases utilizing the prosected specimens provided and discuss the accompanying answers at the end of each case.
Results:
Ninety-four per cent of the students who participated in this study agreed that the case-based review sessions provided a helpful overview of musculoskeletal anatomy content. Student performance on the open-ended, case-based musculoskeletal examination questions showed no significant difference in performance on shoulder, hand, hip, thigh, and leg questions. There was, however, a statistically significant decrease in the students' scores on a forearm question in 2021 compared to 2019.
Conclusion:
This paper describes our integrated, collaborative musculoskeletal course, including case-based review sessions, which was positively received by students as having value in reviewing the musculoskeletal content though it was not found to improve examination performance.
... Red herrings can be added to create garden path scenarios, thus allowing learners to discriminate between similar ailments, and recognize the need for supplemental information through labs or other means to produce a differential diagnosis. This may help learners to become cognizant of what they do not know which is fundamentally challenging (Eva et al., 2004). AR can aid in allowing learners to recognize gaps in their information or knowledge, and then relevant and appropriate means to fill those gaps can be utilized. ...
It is necessary to educate medical students to prepare them for the healthcare world they will enter upon completion of their training. This pertains to not only the content of the education, but also the experiences during the instruction, and the application of the learning. However, during their time in training, a student will likely not encounter all the ailments that they will while working. This means that their education should be supplemented to help them be better prepared. We propose augmented reality (AR) virtual patients as a method to assist in the delivery of this supplementation. This paper’s focus is on enhancing the usefulness of AR by providing learners with interactions with a variety of virtual patients. The target audience for this topic includes medical educators, medical school students, residents, educators interested in AR, and designers of educational AR systems. The mentioned topics can even apply to continuing education or refresher courses for physicians. A review of aspects that can allow AR to be more immersive and beneficial to those learning about various ailments is provided. Elements of AR for medical education training are discussed to enhance the relevance and applicability of the learning experience. These include presenting multiple simultaneous ailments and the ability to modify AR patient characteristics, which can be accomplished more quickly and with more possibilities than when using manikins.
... Ahol a kritikák "fogást találnak" az eredeti koncepción, az leginkább a pszichológiai, illetve statisztikai megalapozás, illetve az interpretáció (Christopher et al. 2021, Gignac 2022, Mcintosh et al. 2019, Pennycook et al. 2017, Zajenkowski et al. 2020. A kutatások egy másik, jelentős része arra irányul, hogy az egyes tevékenységek vagy képességek közti különbséget vizsgálja a DKH szempontjából (Dunning 2011, Ehrlinger et al. 2008, Eva et al. 2004, Nierenberg & Dahl 2021, vagyis azt, hogy eltérő kihívások esetén eltérően jelentkezik-e (ha jelentkezik egyáltalán) a hatás. Arra mindenképpen rávilágítanak ezek a kritikák, hogy a felszíni jelenség mögött álló mechanizmus, illetve a hatást befolyásoló tényezők feltárása egyelőre befejezetlen, és több, változatos területekről származó empirikus bizonyítékot igényel. ...
A TANULMÁNY CÉLJA Dunning–Kruger-hatásként (DKH) ismert az a jelenség, hogy a rosszabbul teljesítő egyének hajlamosabbak pontatlanabbul és felfelé torzítva becsülni saját teljesítményüket. Jelen kutatás e hatást kívánja mérni marketing kurzusok keretében a magyar felsőoktatásban, amivel bővíti a hazai szakirodalomban eddig, ilyen szempontból elemzett tantárgyak körét. ALKALMAZOTT MÓDSZERTAN Az adatgyűjtés 2021 májusában zajlott 3 alapképzéses szak marketing témájú kurzusain, online zárthelyi dolgozatok keretében (N=160). A hallgatók a dolgozat első és utolsó kérdéseként megbecsülhették az elért pontszámukat. A hatás definíciója alapján feltételeztük, hogy a magasabb pontszámot elérő hallgatók pontosabbak és kevésbé becsülik felül pontszámaikat a dolgozat megírása előtt és után egyaránt. További hipotézisünk szerint a jobb teljesítményűek több információt szereznek saját tudásukról a teszt alatt, ezért náluk a vizsga után átlagosan kisebb a pontszámok felülbecslése és a becslési hiba, mint előtte. Vizsgáltuk még, hogy a DKH egyformán jelenik-e meg a nőknél és a férfiaknál is. A kutatásban elsősorban többváltozós lineáris regressziós modelleket alkalmaztunk. LEGFONTOSABB EREDMÉNYEK Eredményeink alátámasztották, hogy minél gyengébb teljesítményt ért el egy hallgató, annál pontatlanabbul ítélte meg és egyben jobban felül is becsülte várható pontszámát. A jobban teljesítők becslései hajlamosabbak voltak a dolgozat végére optimistábbá és pontatlanabbá válni. Ugyanakkor a nemek nem mutattak összefüggést a vizsgált hatással. GYAKORLATI JAVASLATOK A kutatás eredményei alapján a gyengébben teljesítő hallgatók kevésbé vannak tisztában saját tudásukkal, így nem is érzik annyira szükségét a fokozottabb felkészülésnek. Ez az oktatók számára azt jelenti, hogy az oktatás során az ismeretátadáson túl e hatás csökkentésére, kiküszöbölésére is figyelmet kell fordítani, például a kijavított dolgozatok közös áttekintése révén, különösen, mivel a képzés során szerzett kudarcok a lemorzsolódás egyik okát jelentik.
... Ahol a kritikák "fogást találnak" az eredeti koncepción, az leginkább a pszichológiai, illetve statisztikai megalapozás, illetve az interpretáció (Christopher et al. 2021, Gignac 2022, Mcintosh et al. 2019, Pennycook et al. 2017, Zajenkowski et al. 2020. A kutatások egy másik, jelentős része arra irányul, hogy az egyes tevékenységek vagy képességek közti különbséget vizsgálja a DKH szempontjából (Dunning 2011, Ehrlinger et al. 2008, Eva et al. 2004, Nierenberg & Dahl 2021, vagyis azt, hogy eltérő kihívások esetén eltérően jelentkezik-e (ha jelentkezik egyáltalán) a hatás. Arra mindenképpen rávilágítanak ezek a kritikák, hogy a felszíni jelenség mögött álló mechanizmus, illetve a hatást befolyásoló tényezők feltárása egyelőre befejezetlen, és több, változatos területekről származó empirikus bizonyítékot igényel. ...
A TANULMÁNY CÉLJA Az önvezető járművek (AV) bevezetése forradalmasíthatja a mobilitást. Elterjedésük egyik legmeghatározóbb korlátja azonban társadalmi elfogadásuk. Az utóbbi években egyre nő azon szakcikkek száma melyek az AV-k fogyasztói elfogadását vizsgálják valamely technológiaelfogadási modell felhasználásával, kérdőíves megkérdezéssel. Mindinkább előtérbe kerül ugyanakkor ezen kutatási módszertanok azon releváns korlátja, mely szerint a válaszadók nem rendelkeznek személyes tapasztalattal az AV-ben történő utazást illetően. Tanulmányunk célja egy olyan kutatási módszertan fejlesztése és tesztelése, mely ötvözi a technológiaelfogadás és-használat egyesített elméletét (UTAUT) a conjoint elemzéssel. Célunk, hogy megállapíthassuk, melyek azok a legfontosabb termékjellemzők az AV-k tekintetében, amelyek leginkább befolyásolják a fogyasztókat döntéshozás közben. ALKALMAZOTT MÓDSZERTAN Új termékek fejlesztése során sikeresen alkalmazható módszertan a conjoint elemzés. Kutatásunk során ezért a korábban már validált UTAUT modell független változóit, mint termékjellemzőket azonosítottuk egy teljes profilú conjoint elemzés létrehozására. Primer adatgyűjtésünk során a conjoint kártyákat összesen 202 fő értékelte két lépésben. Elsőként saját preferenciáik szerint sorbarendezték, másodikként tíz pontos Likert skálán értékelték a kapott kártyákat, melyek a termékjellemzők szintjeinek különböző kombinációit jelenítették meg. LEGFONTOSABB EREDMÉNYEK Eredményeinkből kiderül, hogy az UTAUT modell segítségével létrehozott teljes profilú conjoint elemzés sikeresen alkalmazható az önvezető járművekhez kapcsolódó fogyasztói preferenciák vizsgálatára. Fontos ugyanakkor megjegyezni, hogy a Likert skála segítségével történő értékelés során megbízhatóbb eredményeket kaptunk, mint a preferenciasorrend felállítása esetén. Legfontosabb eredményünk, hogy válaszadóink számára a biztonság kiemelkedően a legfontosabb termékjellemző, melyet a funkcionális attribútumok némileg lemaradva, mégis kéz a kézben követnek, legkevésbé fontosak pedig a belső jellemzők. GYAKORLATI JAVASLATOK Eredményeinkkel szeretnénk felhívni a fejlesztők és döntéshozók figyelmét első sorban arra, hogy munkájuk során kiemelkedő helyet kapjon az AV-k biztonsága a széleskörű elterjedés érdekében. Irányadó lehet továbbá, hogy eredményeink alapján egy jól működő és kényelmes AV-ért a fogyasztók hajlandóak lennének magasabb árat fizetni. Köszönetnyilvánítás: Készült a Nemzeti Kutatási, Fejlesztési és Innovációs Hivatal – NKFIH K 137571 azonosító számú K_21 „OTKA" Kutatási témapályázat támogatásával.
... D 'Eon, Sadownik, Harrison, and Nation (2008) suggested that the aggregate retrospective post-then data may be a more useful measure to evaluate the overall impact of a teacher education program rather than to measure impact on individual participants. This approach may be useful for IHEs in lieu of the traditional pre/posttest assessments, particularly where participants may be likely to rate themselves higher on the pre-than the post-test, or provide socially desirable responses (Eva, Cunnington, Reiter, Keane, & Norman, 2004;Lam & Bengo, 2003). Faculty may also consider the retrospective post-then approach as a methodology for evaluating their overall teacher education program. ...
Professional development that focuses on supporting teachers' abilities to work with diverse families is keenly needed. This article outlines three instructional strategies and how they were used with undergraduate students in an inclusive early childhood teacher education program: (a) involving diverse family members as instructional supports; (b) performance of family scenarios; and (c) examining diverse families in popular media. These strategies, undergirded by the importance of critical reflection, have been implemented as part of a statewide preservice teacher education initiative in Iowa. Program evaluation data provide preliminary evidence of their potential to foster the development of necessary cultural dispositions for future teachers to meet the growing diversity among children and families.
... Current research has attempted to establish the correlation of SA accuracy with different factors. For example, both Ross (2006) and Eva, Cunnington, Reiter, Keane, & Norman (2004) mentioned that under-achieving students tend to overestimate their performance. Brantmeier, Vanderplank, & Strube (2012) investigated students' SA accuracy and their proficiency level and found that language proficiency level was positively correlated with SA accuracy. ...
Checklists, as a form of self-assessment, have been provided to students by educators as a way to promote learner autonomy. However, due attention has not been paid to the accuracy of those completed checklists. Through a survey, interviews, and comparison of students’ and researchers’ completed checklists, the research intends to investigate the accuracy of students’ self-assessment of their writing in a Sino-foreign university in China. Suggestions on checklist design and pedagogical implications on how to improve learners’ self-assessment are made.
... practices are significant predictors of Self-Directed Learning (SDL) competencies (11) that are considered an important component of life-long learning and thus, a key competency in medical school curricula (12). Based on the principles of self-directed learning, reflective practice motivates students to do self-assessment, improves critical thinking skills, encourages professionalism, and even communication skills (13). ...
E-portfolios based on reflections and reflective practice enhance self-directed learning. The key components of self-directed learning according to Garrison's model include self-management, self-monitoring, and motivation. The aim of the present study was to explore students' perceptions of utilising learning portfolios as a tool for reflective practice and to evaluate their responses based on Garrison's Model of Self-Directed Learning. The current study was conducted among the second-year pre-clinical students at the College of Medicine and Health Sciences, National University of Science and Technology, Sultanate of Oman. 165 students, enrolled for the Behavioural Science in Medicine course were part of the study. They were given an orientation to reflective practice and portfolio writing. Feedback from students was obtained on completion of the course e-portfolio. A majority of students indicated that completing the e-portfolio enhanced self-directed learning, encouraged self-reflection, improved insight, and contributed towards self-awareness. Participants' responses based on Garrison's Model of Self-Directed Learning indicate above average mean scores across all three domains-self-management, self-monitoring, and motivation. Significant inter-domain correlation was also seen The e-portfolio is an effective modality to translate reflective practice to practical learning in undergraduate medical education. There has been a significant impact on students' self-directed learning, critical thinking, and self-monitoring. Using the e-portfolio is extremely advantageous to the holistic development of students-both in their personal and professional domains. It is thus imperative that activities focussing on reflective practice be introduced into formal curricular delivery in undergraduate medical education.
... Although much effort was put into the execution of this research project, there remain several limitations that should be acknowledged. Firstly, since there is probably a gap between teachers' self-report and real capacity [80,81], further research might consider triangulating data sources observations or testing to increase data objectivity. Secondly, the present study investigates teachers' capacity for Action research and the difficulties they encountered, but it does not cover insights into the forms of support for teacher Action research. ...
Action research (AR), which involve rigorous cycles of implementing and reflecting on educational changes, brings genuine values to language teaching. However, due to several factors related to school cultures and the teacher-researchers, AR does not always live up to its expectations. Generally, previous studies have steered towards instigating an awareness of AR at the expense of obtaining empirical evidence on competencies and conditions for AR. This study examines teachers’ action research competencies and challenges in doing AR. A questionnaire (70 items, α = 0.809) was administered to 105 EFL teachers in various high schools in Vietnam. Among these participants, eight partook in follow-up semi-structured interviews. Chi-square analysis confirmed that top-rated competencies aligned with the traditional roles of high school English teachers, including giving feedback, using visual aids for presentations, and self-assessment. In contrast, teachers were least sure of their research skills and techniques. Thematic analysis complemented the quantitative data, ascribing the underdeveloped research capacity to a lack of guidelines, motivation, and resources. Consequently, the practice of AR might be improved with the cultivation of a teacher research culture. Other measures to incentivize high school EFL teachers in research engagement are also discussed in this paper.
... A harmadik hipotézis feltételezi, hogy a felkészültebb hallgatók hajlamosabbak korrigálni önértékelésüket a megszerzett tapasztalatok alapján (ld. Nierenberg & Dahl 2021, Edwards et al. 2003, Eva et al. 2004. Esetünkben ez azt jelenti, hogy számíthatunk arra, hogy a vizsga után azok javítanak jelentősebb mértékben a vizsga előtt tett önértékelésükön, akik jobban teljesítenek. ...
A TANULMÁNY CÉLJA A tanulmány célja Dunning–Kruger-hatás elemzése az üzleti felsőoktatási szakképzésben, illetve annak vizsgálata, hogy az önértékelés pontossága fejleszthető-e a felkészültség emelésével. ALKALMAZOTT MÓDSZERTAN Feleltválasztásos vizsgatesztek eredményeit (N = 359) vetettük össze a hallgatók saját vizsga előtti és utáni becsléseivel, lineáris regressziós modellekben, kontroll változók szerepeltetése mellett. LEGFONTOSABB EREDMÉNYEK A gyengébb teljesítményű hallgatók hajlamosabbak saját teljesítményüket felül- és egyben pontatlanabbul értékelni. A felkészültebb hallgatók hajlamosabbak korrigálni önértékelésüket a megszerzett tapasztalatok alapján. Ezt mind a vizsgák alatt, mind két vizsga között alátámasztottuk. Mindezekben nem találtunk különbséget a nemek között. GYAKORLATI JAVASLATOK Eredményeink alapján mind a tesztírás maga, mind az alaposabb felkészülés csökkentette az önértékelési hibákat, ezért javasolható a tesztelés gyakoriságának növelése (évközi és/vagy próbatesztekkel). Köszönetnyilvánítás: A tanulmány az EFOP3.6.3-VEKOP-16-2017-00007-"Tehetségből fiatal kutató" – A kutatói életpályát támogató tevékenységek a felsőoktatásban projekt támogatásával készült.
... Although studentdirected learning has many strengths, [23,24] it does pose unique challenges to the integration of health humanities within a medical school curriculum. Many students come to medical school with a biomedical background and have limited exposure to the humanities in their undergraduate education. ...
Introduction: The humanities enrich and transform the practice of medicine. What remains to be seen, however, is how best to integrate humanities into the medical curriculum to optimize both educational and patient-related outcomes. The present study considers the structure of an innovative student-driven humanities curriculum and seeks to understand its strengths and limitations, as well as make recommendations for improvement. Results: 420 HLOs met criteria for analysis and were coded as instrumental (developing direct clinical skill), non-instrumental (non-skill based), or both. Of these, 125 (30%) were instrumental, 239 (57%) were non-instrumental, and 56 (13%) were coded as both. Most instrumental HLO centered around communication skills. Non-instrumental HLO most commonly focused on bearing witness and critiquing a particular experience within a social and/or political context. Conclusions: Findings from this study contribute to the development of a humanities curriculum in a student-directed learning program. Non-instrumental HLO lacked a theoretical framework to guide student's investigations to a deeper level of analysis. Student-directed learning offers many strengths, but can be enhanced through external direction from humanities trained faculty, particularly given that many medical students have a limited humanities background.
... While medical students themselves recognize the importance of science-based decision-making, they often do not feel confident in their ability to do so (Pruskil et al., 2009) although there is also evidence of overconfidence effects (Borracci and Arribalzaga, 2018). This, in turn, may be due to deficits in their ability to evaluate their own knowledge and recognize deficiencies therein (Kruger and Dunning, 1999;Eva et al., 2004). With regard to the learning progressions of medical students, making professional decisions and solving problems in clinical contexts require not only the acquisition of knowledge but also an increase in confidence in their medical knowledge (Khan et al., 2001). ...
... [3][4][5][6] Some authors suggest that self-assessment may even be harmful to learning if it is uninformed and uncalibrated by an external assessor. 7,8 Others have suggested that the inconsistencies between learner selfassessment and teacher judgements of competence make it inadvisable to use learner self-assessments in decision-making about competence. 9 For these reasons, it is important that learner self-assessments of progress be monitored for both accuracy (ie, consistency with assessors) and change over the course of training. ...
Background and objectives:
In competency-based medical education (CBME), should resident self-assessments be included in the array of evidence upon which summative progress decisions are made? We examined the congruence between self-assessments and preceptor assessments of residents using assessment data collected in a 2-year Canadian family medicine residency program that uses programmatic assessment as part of their approach to CBME.
Methods:
This was a retrospective observational cohort study using a learning analytics approach. The data source was archived formative workplace-based assessment forms (fieldnotes) stored in an online portfolio by family medicine residents and preceptors. Data came from three academic teaching sites over 3 academic years (2015-2016, 2016-2017, 2017-2018), and were analyzed in aggregate using nonparametric tests to evaluate differences in progress levels selected both within and between groups.
Results:
In aggregate, first-year residents' self-reported progress was consistent with that indicated by preceptors. Progress level rating on fieldnotes improved over training in both groups. Second-year residents tended to assign themselves higher ratings on self-entered assessments compared with those assigned by preceptors; however, the effect sizes associated with these findings were small.
Conclusions:
Although we found differences in the progress level selected between preceptor-entered and resident-entered fieldnotes, small effect sizes suggest these differences may have little practical significance. Reasonable consistency between resident self-assessments and preceptor assessments suggests that benefits of guided self-assessment (eg, support of self-regulated learning, program efficacy monitoring) remain appealing despite potential risks.
... Although student self-reports can be useful for characterizing a wide range of subjective qualities (Lopatto et al., 2008;Cuthbert et al., 2012;Shaffer et al., 2014), recent work has indicated self-assessments of learning do not always correspond with empirical measures, with respondents sometimes overestimating or even underestimating their improvements (Eva et al., 2004;Schiekirka et al., 2013;Ziegler and Montplaisir, 2014). In this study, the application of more rigorous methods such as an established science process skills test (Burns et al., 1985;Dirks and Cunningham, 2006;Feyzíoglu, 2012;Kramer et al., 2018) was avoided, because the endeavor would have entailed the laborious process of translation and validation. ...
Infusing undergraduate curricula with authentic research training is an important contemporary challenge. Such exposure typically occurs through mentored research (MR) or course-based undergraduate research experiences (CUREs). In Asian contexts, CURE implementation is rare, while MR is often a graduation requirement. In this study, mentor interviews and mentee focus groups were used to characterize the learning challenges associated with this requirement at a Chinese university. An intensive 6-week CURE was then implemented as an MR preparatory program to help mitigate the identified challenges. This program contained seven site-specific features not typically included in other CUREs, each designed to improve different aspects of student readiness for MR. Post-CURE surveys, focus groups, and interviews demonstrated CURE enrollment significantly improved subsequent MR outcomes. Almost 90% of all enrollees, for example, began their first MR experience in their second year, more than twice the rate of non-enrollees. Enrollees also reported greater confidence in their research skills and more frequent experiences working in multiple labs. This study reports both immediate CURE and downstream MR outcomes, using the former to help explain the latter. A comprehensive CURE implementation process is described, offering a potential model for the design of other programs with similar research enhancement goals.
... Moreover, as demonstrated in this study, self-reported competence is weakly related to performance-based competence outcomes. Consistent with other work on competency, the relationship between self-evaluated competence and performance-based competence is minimal (Davis et al., 2006;Eva et al., 2004). As such, focus on self-awareness and self-knowledge in competency development (Kaslow et al., 2018) would benefit from ensuring trainee perceptions of their competency align with benchmarked progression. ...
Objective:
Attaining competence in assessment is a necessary step in graduate training and has been defined to include multiple domains of training relevant to this attainment. While important to ensure trainees meet these standards of training, it is critical to understand how and if competence shapes a trainees' professional identity, therein promoting lifelong competency.
Methods:
The current study assessed currently enrolled graduate trainees' knowledge and perception of their capabilities related to assessment to determine if self-reported and performance-based competence would incrementally predict their intention to use assessment in their future above basic training characteristics and intended career interests.
Results:
Self-reported competence, but not performance-based competence, played an incremental role in trainees' intention to use assessments in their careers. Multiple graduate training characteristics and practice experiences were insignificant predictors after accounting for other relative predictors (i.e., intended career settings, integrated reports).
Conclusion:
Findings are discussed about the critical importance of incorporating a hybrid competency-capability assessment training framework to further emphasize the role of trainee self-efficacy in hopes of promoting lifelong competence in their continued use of assessments.
... Therefore, our participants' narratives about the aspects of training that were most beneficial for their learning should be contextualized within the literature on the limitations of self-assessment. 40 ...
Introduction:
Effective leadership improves patient care during medical and trauma resuscitations. While dedicated training programs can improve leadership in trauma resuscitation, we have a limited understanding of the optimal training methods. Our objective was to explore learners' and teachers' perceptions of effective methods of leadership training for trauma resuscitation.
Methods:
We performed a qualitative exploration of learner and teacher perceptions of leadership training methods using a modified grounded theory approach. We interviewed 28 participants, including attending physicians, residents, fellows, and nurses who regularly participated in trauma team activations. We then analyzed transcripts in an iterative manner to form codes, identify themes, and explore relationships between themes.
Results:
Based on interviewees' perceptions, we identified seven methods used to train leadership in trauma resuscitation: reflection; feedback; hands-on learning; role modeling; simulation; group reflection; and didactic. We also identified three major themes in perceived best practices in training leaders in trauma resuscitation: formal vs informal curriculum; training techniques for novice vs more senior learner; and interprofessional training. Participants felt that informal training methods were the most important part of training, and that a significant part of a training program for leaders in trauma resuscitation should use informal methods. Learners who were earlier in their training preferred more supervision and guidance, while learners who were more advanced in their training preferred a greater degree of autonomy. Finally, participants believed leadership training for trauma resuscitation should be multidisciplinary and interprofessional.
Conclusion:
We identified several important themes for training leaders in trauma resuscitation, including using a variety of different training methods, adapting the methods used based on the learner's level of training, and incorporating opportunities for multidisciplinary and interprofessional training. More research is needed to determine the optimal balance of informal and formal training, how to standardize and increase consistency in informal training, and the optimal way to incorporate multidisciplinary and interprofessional learning into a leadership in trauma resuscitation training program.
... While medical students themselves recognize the importance of science-based decision-making, they often do not feel confident in their ability to do so (Pruskil et al., 2009) although there is also evidence of overconfidence effects (Borracci and Arribalzaga, 2018). This, in turn, may be due to deficits in their ability to evaluate their own knowledge and recognize deficiencies therein (Kruger and Dunning, 1999;Eva et al., 2004). With regard to the learning progressions of medical students, making professional decisions and solving problems in clinical contexts require not only the acquisition of knowledge but also an increase in confidence in their medical knowledge (Khan et al., 2001). ...
Research Focus: The promotion of domain-specific knowledge is a central goal of higher education and, in the field of medicine, it is particularly essential to promote global health. Domain-specific knowledge on its own is not exhaustive; confidence regarding the factual truth of this knowledge content is also required. An increase in both knowledge and confidence is considered a necessary prerequisite for making professional decisions in the clinical context. Especially the knowledge of human physiology is fundamental and simultaneously critical to medical decision-making. However, numerous studies have shown difficulties in understanding and misconceptions in this area of knowledge. Therefore, we investigate (i) how preclinical medical students acquire knowledge in physiology over the course of their studies and simultaneously gain confidence in the correctness of this knowledge as well as (ii) the interrelations between these variables, and (iii) how they affect the development of domain-specific knowledge.
Method: In a pre–post study, 169 medical students’ development of physiology knowledge and their confidence related to this knowledge were assessed via paper-pencil questionnaires before and after attending physiology seminars for one semester. Data from a longitudinal sample of n = 97 students were analyzed using mean comparisons, regression analyses, and latent class analyses (LCAs). In addition, four types of item responses were formed based on confidence and correctness in the knowledge test.
Results: We found a significant and large increase in the students’ physiology knowledge, with task-related confidence being the strongest predictor (apart from learning motivation). Moreover, a significantly higher level of confidence at t2 was confirmed, with the level of prior confidence being a strong predictor (apart from knowledge at t2). Furthermore, based on the students’ development of knowledge and confidence levels between measurement points, three empirically distinct groups were distinguished: knowledge gainers, confidence gainers, and overall gainers. The students whose confidence in incorrect knowledge increased constituted one particularly striking group. Therefore, the training of both knowledge and the ability to critically reflect on one’s knowledge and skills as well as an assessment of their development in education is required, especially in professions such as medicine, where knowledge-based decisions made with confidence are of vital importance.
... Moreover, we decided to limit the number of questions asked to increase the number of participants. As self-assessments can be an unreliable proxy of actual knowledge (29), some of our survey questions may be prone to self-evaluation bias. In addition, data collection took place mostly at the beginning of the pandemic. ...
Background: The emerging field of artificial intelligence (AI) will probably affect the practice for the next generation of doctors. However, the students' views on AI have not been largely investigated.
Methods: An anonymous electronic survey on AI was designed for medical and dental students to explore: (1) sources of information about AI, (2) AI applications and concerns, (3) AI status as a topic in medicine, and (4) students' feelings and attitudes. The questionnaire was advertised on social media platforms in 2020. Security measures were employed to prevent fraudulent responses. Mann-Whitney U-test was employed for all comparisons. A sensitivity analysis was also performed by binarizing responses to express disagreement and agreement using the Chi-squared test.
Results: Three thousand one hundred thirty-three respondents from 63 countries from all continents were included. Most respondents reported having at least a moderate understanding of the technologies underpinning AI and of their current application, with higher agreement associated with being male (p < 0.0001), tech-savvy (p < 0.0001), pre-clinical student (p < 0.006), and from a developed country (p < 0.04). Students perceive AI as a partner rather than a competitor (72.2%) with a higher agreement for medical students (p = 0.002). The belief that AI will revolutionize medicine and dentistry (83.9%) with greater agreement for students from a developed country (p = 0.0004) was noted. Most students agree that the AI developments will make medicine and dentistry more exciting (69.9%), that AI shall be part of the medical training (85.6%) and they are eager to incorporate AI in their future practice (99%).
Conclusion: Currently, AI is a hot topic in medicine and dentistry. Students have a basic understanding of AI principles, a positive attitude toward AI and would like to have it incorporated into their training.
... Given the significant global changes and move to remote learning, how these results generalize and how providers' preferences have already changed is unknown. Finally, our participants' preferences should be viewed in the context of the literature on the limitations of self-assessment and judgments of learning [14]. As such, these findings should guide future studies rather than be viewed as guidance on optimizing CME delivery. ...
Study objective
We investigate the continuing medical education delivery preferences among emergency medicine providers, both physicians and advanced practice providers (APPs), within a large, national emergency medicine group.
Methods
A survey was sent via email to all emergency medicine health care providers in the practice group, including questions evaluating both delivery method and topic preference for continuing education. The study was sent to providers whom the group employed from October 2019 through January 2020.
Results
A total of 2038 providers, 1373 physicians, and 665 APPs completed the survey - a completion rate of 68.7%. In general, practitioners expressed willingness to learn across various platforms, with a strong overall preference towards online and on-demand options, including video, podcast, and written materials. Universally, a larger percentage of APPs identified a desire for more continuing education across all topics than physicians.
Conclusion
Education preferences among emergency medicine providers vary with a strong trend toward online and on-demand content. Understanding the delivery and topic preferences for providers is important for the optimal creation of continuing education content.
... Given that humans are notoriously bad at self-assessment, maintaining appropriate levels of confidence can be challenging if left entirely to the individual to resolve. [21][22][23]51,52 Rather than measuring confidence as an independent variable or construct, we can consider it as being linked to (but not as a surrogate for) competence. The interaction of confidence and competence in HPE has long been considered, [53][54][55][56][57][58] and individuals are viewed to be problematic when confidence and competence are decoupled. ...
Context
There have been significant advances in competency-based medical education (CBME) within health professions education. While most of the efforts have focused on competency, less attention has been paid to the role of confidence as a factor in preparing for practice. This paper seeks to address this deficit by exploring the role of confidence and the calibration of confidence with regard to competence.
Methods
This paper presents a conceptual review of confidence and the calibration of confidence in different medical education contexts. Building from an initial literature review, the authors engaged in iterative discussions exploring divergent and convergent perspectives, which were then supplemented with targeted literature reviews. Finally, a stakeholder consultation was conducted to situate and validate the provisional findings.
Results
A series of axioms were developed to guide perceptions and responses to different states of confidence in health professionals: 1) confidence can shape how we act and is optimized when it closely corresponds to reality; 2) self-confidence is task-specific, but also inextricably influenced by the individual self-conceptualization, the surrounding system, and society; 3) confidence is shaped by many external factors and the context of the situation; 4) confidence must be considered in conjunction with competence; and 5) the confidence-competence ratio (CCR) changes over time. It is important to track learners’ CCRs and work with them to maintain balance.
Conclusion
Confidence is expressed in different ways and is shaped by a variety of modifiers. While CBME primarily focuses on competency, proportional confidence is an integral component in ensuring safe and professional practice. As such, it is important to consider both confidence and competence, as well as their relationship in CBME. The CCR can serve as a key construct in developing mindful and capable health professionals. Future research should evaluate strategies for assessing CCR, identify best practices for teaching confidence and guiding self-calibration of CCR, and explore the role of CCR in continuing professional development for individuals and teams.
... clinical data) could be digitalized and integrated with these data to further support growth. These data are important because self-assessment without this information often leads to erroneous conclusions (Kruger and Dunning 1999;Eva et al. 2004;Regehr 2005, 2008;Regehr and Eva 2006;Mann et al. 2011). Data are required to identify weaknesses, destabilize selfconceptualizations, flatten hierarchies, identify gaps between perceptions and reality, and alter developmental timelines. ...
Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO’s conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.
... 23,24 The finding that residents were significantly less confident in their learning while driving, despite no difference in assessment of recall, is consistent with the known limitations of self-assessment and judgment of learning. [25][26][27] Residents should be aware that their perceptions of where learning best takes place may not be accurate. Further, they should be aware that recall of podcast knowledge over time is significantly impaired (>20% drop in test scores after 1 month). ...
Study objective
Emergency medicine residents use podcasts as part of their learning process, often listening while driving. It is unclear how driving while listening to a podcast affects knowledge acquisition and retention. This study evaluated the knowledge gained from listening to podcasts while driving compared to that gained from undistracted listening.
Methods
This was a multicenter, randomized, crossover trial among postgraduate year (PGY) 1 to 4 emergency medicine residents at 4 institutions. Residents were randomized with stratification by site and PGY level to listen to podcasts while driving first or sitting undistracted in a room first. Within 30 minutes of listening, they completed a 20-question test. They subsequently crossed over to the alternate intervention, serving as their own controls, and listened to a different podcast before completing a second 20-question test. Each of the podcasts was professionally recorded and based on 5 emergency medicine-relevant journal articles that had not been covered in a journal club or curriculum at any of the institutions. One month later, participants completed a delayed recall test composed of 40 new questions based on both podcasts. Questions were derived and validity evidence was collected prior to use. Data were compared using a paired-sample t test and ANOVA.
Results
A total of 100 residents completed the initial recall tests, and 96 residents completed the delayed recall test. There was no statistically significant difference between the driving and undistracted cohorts on the initial recall (74.2% versus 73.3%) or delayed recall (52.2% versus 52.0%).
Conclusion
Driving while listening to a podcast does not meaningfully affect knowledge acquisition or retention when compared with undistracted podcast listening among emergency medicine residents.
... One benefit of this change is that experienced clinicians often recognize similar situations (e.g., familiar problems) more quickly [68]. Importantly, experience may not improve their ability to adapt commonly used strategies to resolve new problems or increase their ability to recognize errors in their own thinking processes [50,[69][70][71]. Since heuristics will always have a role in decision-making due to the speed expected in real-time interactions, it is important to understand what information is used in heuristic decision-making. ...
Significant efforts in the past decades to teach evidence-based practice (EBP) implementation
has emphasized increasing knowledge of EBP and developing interventions to support adoption
to practice. These efforts have resulted in only limited sustained improvements in the daily use
of evidence-based interventions in clinical practice in most health professions. Many new interventions
with limited evidence of effectiveness are readily adopted each year—indicating openness to
change is not the problem. The selection of an intervention is the outcome of an elaborate and complex cognitive process, which is shaped by how they represent the problem in their mind and is mostly invisible processes to others. Therefore, the complex thinking process that supports the appropriate adoption of interventions should be taught more explicitly. Making the process visible to clinicians increases the acquisition of the skills required to judiciously select one intervention over others. The purpose of this paper is to provide a review of the selection process and the critical analysis that is required to appropriately decide to trial or not trial new intervention strategies with patients.
... These were designed for a specific skill or purpose with more defined instruction compared to SDL directed at a wider-focused subject. SDL is considered an important component of life-long learning and is a key competency in medical school curricula (Eva et al., 2004). However relevant skills for SDL should be taught to ensure participants can successfully engage with these activities rather than assuming they are inherent (Gaines et al., 2018). ...
Introduction
As traditional clinical teaching faces major obstacles during the COVID-19 pandemic, medical educators look toward remote teaching methods to provide solutions to allow continuation of teaching. Remote methods, teaching delivered other than face-to-face, align with the transformation seen within pedagogy over the last 20 years.
Aim
The aim of this scoping review was to i) identify existing teaching methods available to remotely teach cardiovascular knowledge or skills and ii) identify if they have been evaluated.
Methods
A scoping review of the literature was undertaken to synthesise available evidence and examine remote teaching methods for application to undergraduate medical education.
Results
Forty-two articles were identified which presented remote teaching methods using either teaching based online, computer-programs, digital resources, mobile-phone technology, podcasts, serious gaming, social media or resources to aid self-directed learning. Although results were heterogenous, they gave an indication of the method’s usefulness. However, evaluations were not consistent and if they were, would have strengthened the value of the findings.
Conclusion
Various remote teaching methods are available to replace face-to-face cardiovascular teaching where this is not possible. Evidence for effectiveness and engagement of individual platforms are variable. Given the ongoing COVID-19 pandemic, medical educators should prioritise ongoing evaluation of remote teaching methods and share best practice.
... One benefit of this change is that experienced clinicians often recognize similar situations (e.g., familiar problems) more quickly [68]. Importantly, experience may not improve their ability to adapt commonly used strategies to resolve new problems or increase their ability to recognize errors in their own thinking processes [50,[69][70][71]. Since heuristics will always have a role in decision-making due to the speed expected in real-time interactions, it is important to understand what information is used in heuristic decision-making. ...
Significant efforts in the past decades to teach evidence-based practice (EBP) implementation has emphasized increasing knowledge of EBP and developing interventions to support adoption to practice. These efforts have resulted in only limited sustained improvements in the daily use of evidence-based interventions in clinical practice in most health professions. Many new interven-tions with limited evidence of effectiveness are readily adopted each year - indicating openness to change is not the problem. The selection of an intervention is the outcome of an elaborate and complex cognitive process which is shaped by how they represent the problem in their mind and is mostly invisible processes to others. Therefore, the complex thinking process which support appropriate adoption of interventions should be taught more explicitly. Making the process visible to clinicians increases the acquisition of the skills required to judiciously select one in-tervention over others. The purpose of this paper is to provide a review of the selection process and the critical analysis that is required to appropriately decide to trial or not trial new intervention strategies with patients.
... The assessment was driven by the demand for constructive adjustment between intentions and evaluations of PBL, the conventional summative assessment model was not really t well to PBL [15]. Hence, the formative assessment protocols including student performance, student self-mutual evaluation and tutor teaching evaluation were adopted in this study, which provide unique opportunity to judge each other's work between tutors and students wherefore making them develop the ability to re ect on own strengths and weaknesses as these are central elements of self-directed learning (SDL) [16]. ...
Objectives: Evaluate the effectiveness of PBL teaching and analyze potential influencing factors between senior and junior from the perspectives of student performance, student self-mutual and tutor teaching evaluations.
Results: Study found the overall three comparative assessments scores of Grade 4 were significantly higher than those of Grade 1(P＜0.01). Further analysis revealed no significant difference of student performance score between Grade 4 and Grade 1 in the 1st unit (P＞0.05), but it was improved and maintained significantly since 2nd time (P＜0.01).Moreover, each self-mutual evaluation of 4th year students was significantly higher than that of 1st year students, especially in the second half of the course (P＜0.01).And, all the evaluations of tutor teaching from seniors were significantly higher than those from juniors (P＜0.01). Finally, no significant correlation between student self-mutual scores and those given by tutors no matter high or lower grade (P＞0.05).
... Our representations of residents' perceptions of effectiveness must be contextualized within the literature on the limitations of selfassessment and judgments of learning. [50][51][52] These findings should serve as a guide to future realist evaluations that examines these features to determine what works, where, and why, 53 rather than be viewed as a definitive prescription for podcast optimization. ...
Introduction:
Educational podcasts are used by emergency medicine (EM) trainees to supplement clinical learning and to foster a sense of connection to broader physician communities. Yet residents report difficulties remembering what they learned from listening, and the features of podcasts that residents find most effective for learning remain poorly understood. Therefore, we sought to explore residents' perceptions of the design features of educational podcasts that they felt most effectively promoted learning.
Methods:
We used a qualitative approach to explore EM trainees' experiences with educational podcasts, focusing on design features that they found beneficial to their learning. We conducted 16 semi-structured interviews with residents from three institutions from March 2016-August 2017. Interview transcripts were analyzed line-by-line using constant comparison and organized into focused codes, conceptual categories, and then key themes.
Results:
The five canons of classical rhetoric provided a framework for thematically grouping the disparate features of podcasts that residents reported enhanced their learning. Specifically, they reported valuing the following: 1) Invention: clinically relevant material presented from multiple perspectives with explicit learning points; 2) Arrangement: efficient communication; 3) Style: narrative incorporating humor and storytelling; 4) Memory: repetition of key content; and 5) Delivery: short episodes with good production quality.
Conclusion:
This exploratory study describes features that residents perceived as effective for learning from educational podcasts and provides foundational guidance for ongoing research into the most effective ways to structure medical education podcasts.
... 17 Thus, as overall resources become less available, programs may be forced to prioritize improvements based on a variety of factors including evolving expectations from accrediting bodies, changes in the pharmacy practice environment, as well as shifts in the recognition of the importance of enhancing student and faculty personal well-being and professional development. 18,19 As pharmacy programs aim to provide high quality education experiences while meeting accreditation expectations, more information is needed to better understand successful and sustainable co-curriculum. Given the dynamic environment surrounding accreditation requirements and lack of consistent definitions related to the co-curriculum, our group sought to identify and describe the barriers facing pharmacy programs in implementing the co-curriculum in their programs and evaluate programs' confidence in meeting the new co-curriculum ACPE requirement. ...
Objective. To determine areas of concern, and challenges to implementing and assessing the co-curriculum in accredited Doctor of Pharmacy programs, along with how confident programs are in their ability to meet the co-curriculum requirement as mandated by the Accreditation Council for Pharmacy Education (ACPE). Methods. A survey was administered to all ACPE-accredited pharmacy programs to collect information regarding areas of concern, challenges, and confidence in their ability to meet the co-curriculum requirement. The frequency of responses to items are presented along with comparisons based on characteristics, including institution type, cohort size, most recent ACPE accreditation review, and supporting offices. Results. The most common concerns centered on the documentation and assessment process. The most commonly reported challenges were lack of enthusiasm or buy-in from faculty, staff, and students; lack of a clear definition of co-curriculum; and faculty time and insufficient staff. Overall, programs had a high level of confidence in their ability to meet the requirements for co-curriculum. The only differences found were related to supporting offices and cohort size. Conclusion. The results suggest that having supporting offices may reduce the co-curriculum burden. Similarly, student cohort size may have an impact on the challenges for some programs, particularly those with moderate-sized cohorts reporting challenges related to faculty and staff. Further research is needed to determine how programs address these critical issues, and to explore whether programs report differently on these areas after completing an accreditation review. The study results may be useful to members of the Academy when evaluating co-curriculum.
... Most important, a belief in self-efficacy is not itself a measure of effectiveness. Research has shown, repeatedly and across disciplines, that self-assessments inflate communication competence relative to external evaluation (e.g., Duran & Zakahi, 1987;Eva et al., 2004;Gruppen et al., 1997;McCroskey & McCroskey, 1988;Mort & Hansen, 2010). The key measure of the effectiveness of any form of communication training is not only evidence that a target audience judges the trainee effective (Bray, 2012;Rodgers et al., 2018) but also that the target audience finds the trainee a more effective communicator after training than before. ...
As the science community has recognized the vital role of communicating to the public, science communication training has proliferated. The development of rigorous, comparable approaches to assessment of training has not kept pace. We conducted a fully controlled experiment using a semester-long science communication course, and audience assessment of communicator performance. Evaluators scored the communication competence of trainees and their matched, untrained controls, before and after training. Bayesian analysis of the data showed very small gains in communication skills of trainees, and no difference from untrained controls. High variance in scores suggests little agreement on what constitutes “good” communication.
... Although the Accreditation Council for Graduate Medical Education recommends that residents complete self-assessments periodically, 4 studies have demonstrated that self-assessments by medical students, interns, and residents in general surgery, family medicine, and psychiatry are invalid measures of their objective performance. [5][6][7][8][9] Several medical studies support different portions of the Dunning-Kruger Curve. For example, 1 study found that junior physicians who were less competent tended to self-rate their efficacy higher than it was demonstrated (point A). 10 Comparison of selfassessments and peer assessments of physicians in internal medicine, pediatrics, and psychiatry showed that those assessed in the lowest quartile by their colleagues tended to rate themselves 30 to 40 percentile ranks higher than their peers. ...
... Am I about to do the right thing?), whilst self-assessment refers to a judgment on one's cumulative evaluation of overall performance (e.g. do I need to refresh my understanding of the best practice for this?) [2,3,5]. ...
Background:
Self-monitoring is an important component of clinical practice. It is underpinned by the framework of self-efficacy which is concerned with judgments of how well one believes one can achieve or perform a task. This research aimed to develop criteria for adequate self-monitoring, then to measure patterns of self-monitoring, and to explore how these patterns relate to a student's year in a medical course and to patterns of knowledge.
Methods:
Analysis of individuals' levels of correctness in answering assessment items and their certainty in correctness may be used to inform assessments of ability to self-monitor. Two criteria were proposed and applied to define adequate self-monitoring. Firstly, increasing proportions correct with increasing levels of certainty. Secondly, having a proportion correct for high certainty responses that was not lower than cohort levels. Student responses in progress tests comprising multiple-choice questions (MCQs) and associated certainty were analysed. Criteria for the presence of adequate self-monitoring and for adequate knowledge were applied to the results of each of four tests conducted over 2 years, and used to categorise patterns of self-monitoring and knowledge.
Results:
Data from 3 year group cohorts totalling 737 students were analysed. The majority (58%) of students demonstrated adequate knowledge and met both criteria for adequate self-monitoring across all four tests. The most advanced year group cohort had the highest rates of adequate knowledge and the highest rates of meeting both self-monitoring criteria. The patterns of self-monitoring were the same as the patterns of knowledge across the four tests for 454 students, but for the remaining 283 the patterns of self-monitoring and knowledge differed.
Conclusion:
Analysis of responses to item level certainty has informed development of a definition of adequate self-monitoring that may be applied to individual student's responses from a single test, and to track the adequacy of a student's self-monitoring over time. Patterns of self-monitoring tend to match patterns of knowledge, but not in all cases, suggesting the self-monitoring measure could provide additional information about student ability.
Confusion is the primary epistemic emotion in the learning process, influencing students' engagement and whether they become frustrated or bored. However, research on confusion in learning is still in its early stages, and there is a need to better understand how to recognize it and what EEG signals indicate its occurrence. The present work investigates confusion during reasoning learning using EEG, and aims to fill this gap with a multidisciplinary approach combining educational psychology, neuroscience and computer science.
Approach: First, we design an experiment to actively and accurately induce confusion in reasoning. Second, we propose a subjective and objective joint labeling technique to address the label noise issue. Third, to confirm that the confused state can be distinguished from the non-confused state, we compare and analyze the mean band power of confused and unconfused states across five typical bands. Finally, we present an EEG database for confusion analysis, together with benchmark results from conventional (Naive Bayes, SVM, Random Forest, and ANN) and end-to-end (LSTM, ResNet, and EEGNet) machine learning methods.
Main results: Findings revealed: 1. Significant differences in the power of delta, theta, alpha, beta and lower gamma between confused and non-confused conditions; 2. A higher attentional and cognitive load when participants were confused; and 3. The Random Forest algorithm with time-domain features achieved a high accuracy/F1 score (88.06%/0.88 for the subject-dependent approach and 84.43%/0.84 for the subject-independent approach) in the binary classification of the confused and non-confused states.
Significance: The study advances our understanding of confusion and provides practical insights for recognizing and analyzing it in the learning process. It extends existing theories on the differences between confused and non-confused states during learning and contributes to the cognitive-affective model. The research enables researchers, educators, and practitioners to monitor confusion, develop adaptive systems, and test recognition approaches.
.
Introduction
Selecting medical students for residency is a competitive process, with a narrow range of scores separating middle-ranked applicants. Self-assessment is a fundamental skill for any competent physician with a demonstrated correlation to diagnostic ability, examination scores, and technical skills, but has yet to be investigated in residency selection. The objective of this study was to investigate the relationship between self-assessment and interview performance as a potential adjunct to discriminate between applicants.
Methods
At the University of Ottawa in 2020, 55 applicants completed a 9-station interview circuit assessing different characteristics or skills important for a career in general surgery, followed by a self-assessment questionnaire evaluating their perceived performance at each station. Pearson's correlation was used to determine the relationship between self-assessment scores (SASs) and interviewer scores (ISs).
Results
There was a negative correlation between SASs and ISs for all interview stations. High performers underestimated their interview performance, and low performers overestimated their performance. Seven of the nine stations reached statistical significance (r = 0.60-0.73, P < 0.001). There was significant variability in the SAS of middle-ranked applicants, with a range three times greater than the range of ISs and demonstrating distinct self-assessment skills in candidates with very similar scores.
Conclusions
Although we strive to select applicants who will succeed in residency to become competent physicians, self-assessment skills may be a useful adjunct during the interview process to assist in discriminating between applicants with similar scores.
Introduction
The objective of this study was to compare faculty member evaluations to student self-evaluations in a clinical endodontic course in the dental school at King Saud University and to evaluate the reliability of the students’ self-assessment scores after using a rubric with well-defined criteria.
Materials and methods
Evaluated and self-evaluated endodontic cases that were clinically treated by fourth-year undergraduate dental students at the College of Dentistry, Girls University Campus (GUC), at King Saud University over two years (2017-2018) were included. Cases included anterior teeth, premolars, and molars. The evaluation form was divided into six sections with well-defined criteria to cover all aspects of nonsurgical root canal treatment with a maximum grade of 10 points can be scored for each student per case. The students evaluated themselves for each section, and then were evaluated by two faculty members. Student and faculty assessment agreement and the reliability of the students’ self-assessment scores were measured. A P-value ≤ 0.05 was considered significant.
Results
A total of 363 cases were included: 26.7% anterior teeth, 38.84% premolars, and 34.43% molars. The students evaluated themselves with higher grades compared to the evaluators’ grading in all steps and in the overall grading in all teeth types. The students’ self-assessment scores showed good and moderate reliability in all steps and in the overall grading.
Conclusion
The students tend to overrate their performance and their assessments have moderate to good reliability, which reflects the reliability of the rubric used as an accurate measurement tool that helps the evaluator and the student objectively assess their performance.
Background
Many medical schools offer a culminating internship readiness experience. Curricula focus on particular knowledge and skills critical to internship, such as answering urgent nursing pages. Studies have shown student performance improvement with mock paging education programs, but the role of feedback versus self-regulated practice has not been studied.Design and Methods
The interprofessional mock paging program included 156 medical students enrolled in a 4th-year internship readiness course and 44 master’s level direct entry nursing students. Medical students were randomized to receive verbal feedback immediately after each of the three phone calls (intervention group) or delayed written feedback (control group) after the third phone call only. Specialty-specific case scenarios were developed and a single checklist for all scenarios was developed using the communication tool ISBAR. Medical students and nursing students had separate training sessions before the pages commenced. The nursing students administered the phone calls and evaluated the medical students by ISBAR checklist. An interrater reliability measure was obtained with physician observation of a selection of phone calls.ResultsAfter adjusting for the case effects (different case scenarios for different specialties), students showed no statistically significant differences on checklist scores for case 1 (first case, F = 1.491, df = 1, p = .224), but did show statistically significant differences on checklist scores for case 3 (final case, F = 12.238, df = 1, p = .001). Strong interrater reliability was found between the faculty physician and observed nursing students (ICC = .89).Conclusions
Immediate feedback significantly improves student checklist scores with a mock paging program. This finding suggests that coaching with feedback may have advantages above self-regulated learning.
Self-assessment, or students’ evaluation of their own learning or performance in academic tasks, is a self-regulatory process that is intertwined with learners’ cognitive, behavioral, and motivational processes. Among the motivational beliefs that have been studied in relation to self-assessment is self-efficacy, which refers to beliefs about one's capability to learn or perform a specified task successfully. The current study describes the development and validation of the Self-Efficacy for Self-Assessment in Argumentative Writing (SEESA-AW) scale to measure college students’ self-efficacy beliefs for self-assessment in the domain of argumentative writing with two samples of undergraduate college students (N=335 and N=662). Exploratory and confirmatory factor analyses revealed three factors (grammar & mechanics, cohesiveness/flow, the presentation of arguments) that explained 65.4% of the variance. Evidence is also provided for convergent validity of the scale with regard to other writing self-efficacy scales. Educational implications are discussed.
Background:
Electromagnetic navigation bronchoscopy (ENB) is a relatively new and technically demanding procedure for the guidance of bronchoscopic biopsy to help locate small lung lesions. The results in experienced hands are well described. However, we do not know the results in unexperienced hands-in other words, we have no knowledge about how fast you can learn the procedure.
Aim:
The aim of this study was to draw learning curves for beginners in ENB using Cumulated Sum (CUSUM) analysis, a method for quantitative evaluation of the learning curves for clinical procedures.
Methods:
Four operators from 3 centers without prior ENB experience were enrolled. The outcome of consecutive ENB procedures was assessed and classified as either diagnostic or nondiagnostic based on sample adequacy. Learning curves were drawn based on diagnostic yield.
Results:
A total of 215 procedures were assessed. For 2 of the operators (operators 1 and 4), at least 25 to 30 procedures were necessary to obtain competency whereas operators 2 and 3 showed more horizontal learning curves indicating an overweight of diagnostic procedures from the beginning.
Conclusion:
Operators achieve competences in ENB at different paces. This must be taken in account when beginners start to learn the procedure. There is a huge need for a structured educational program and a validated test to determine competences.
The purpose of this paper is to study the effectiveness of oral presentation as an assessment tool in a Finance subject. Assessment data collected from a postgraduate Finance subject in an Australian university over a period of five years from 2005 to 2009 was analysed statistically to determine the relation between students’ performance in oral presentation and other forms of assessments. The sample consists of assessment records of 412 students and 98 group presentations. From the study of correlations between oral presentations and other assessments, it is concluded that students perform better in written assessments compared to oral assessment. The study of effect of gender on students’ performance leads to the conclusion that female students perform better than male students in all forms of assessments except oral presentations where male students perform better although difference between males and females in oral presentation is not very large. The study of students’ performance based on their nationality leads to the conclusion that domestic students perform better than international students in all forms of assessments. Based on the study of student’ performance in oral presentation, it is found that students did well in the development of content of presentations, quality of their analysis, group coordination and organisation of presentation. There is however a general tendency to treat group work as a sum of parts instead of treating the group work as a single task. This study is limited by the fact that effectiveness of oral presentation is studied in only one Finance subject. This study makes an original contribution to the literature as the effectiveness of oral assessment in Finance subject is being studied for the first time. The conclusions arrived in this paper have many implications for policies and practice of learning and teaching in Finance.
Background
Accurate self-assessment of knowledge and technical skills is key to self-directed education required in surgical training. We aimed to investigate the presence and magnitude of cognitive bias in self-assessment among a cohort of surgical interns.
Methods
First-year general surgery residents self-assessed performance on a battery of technical skill tasks (knot tying, suturing, vascular anastomosis, Fundamentals of Laparoscopic Skills peg transfer and intracorporeal suturing) at the beginning of residency. Each self-assessment was compared to actual performance. Bias and deviation were defined as arithmetic and absolute difference between actual and estimated scores. Spearman correlation assessed covariation between actual and estimated scores. Improvement in participant performance was analyzed after an end-of-year assessment.
Results
Participants (N = 34) completed assessments from 2017 to 2019. Actual and self-assessment scores were positively correlated (0.55, P < .001). Residents generally underestimated performance (bias -4.7 + 8.1). Participants who performed above cohort average tended to assess themselves more negatively (bias -7.3 vs -2.3) and had a larger discrepancy between self and actual scores than below average performers (deviation index 9.7 + 8.2 vs 3.8 + 3.1, P < .05). End-of-year total scores improved in 31 (91.2%) participants by an average of 11 points (90 possible). Least accurate residents in initial self-assessments (deviation indices >75th percentile) improved less than more accurate residents (median 5 vs 16 points, P < .05). All residents with a deviation index >75 percentile underestimated their performance.
Conclusion
Cognitive bias in technical surgical skills is apparent in first-year surgical residents, particularly in those who are higher performers. Inaccuracy in self-assessment may influence improvement and should be addressed in surgical training.
Study objectives:
The referral burden on healthcare systems for routine sleep disorders could be alleviated by educating primary care providers (PCPs) to diagnose and manage patients with sleep health issues. This requires effective professional education strategies and resources. This scoping review examined the literature on existing approaches to educate PCPs in sleep health management.
Methods:
A comprehensive literature search was conducted across eight databases to identify citations describing the education of PCPs in diagnosing and managing sleep disorders, specifically insomnia and sleep apnea. A conceptual framework, developed from the knowledge-to-action cycle, was used to analyze citations from a knowledge translation perspective.
Results:
Searches identified 616 unique citations and after selection criteria were applied, 22 reports were included. Reports spanning 38 years were analyzed using components of the knowledge-to-action cycle to understand how educational interventions were designed, developed, implemented, and evaluated. Interventions involved didactic (32%), active (18%) and blended (41%) approaches, using face-to-face (27%), technology-mediated (45%) and multimodal (5%) delivery. Educational effectiveness was assessed in 73% of reports, most commonly using a pre/post questionnaire (41%).
Conclusions:
While this scoping review has utility in describing existing educational interventions to upskill PCPs to diagnose and manage sleep disorders, the findings suggest that interventions are often developed without explicitly considering the evidence of best educational practice. Future interventional designs may achieve greater sustained effectiveness by considering characteristics of the target audience, the pedagogical approaches best suited to its needs, and any environmental drivers and barriers that might impede the translation of evidence into practice.
Purpose:
Objective of this study was to determine if perceptual abilities test (PAT), quantitative reasoning (QR), and academic average (AA) scores predict performance in a rudimentary dental anatomy laboratory exercise.
Methods:
First-year dental students (n = 88) completed two identical wax carving exercises during the first and last weeks of the dental anatomy course. After being given detailed instructions and a wax block, students carved a cube and a semilunar shape using the wax subtraction technique. Using a rubric, students self-assessed their work as unsatisfactory, satisfactory, or ideal in the following areas: finish, sharpness, symmetry, and accuracy. Two calibrated and blinded faculty graded the exercise using the same rubric as the students. Cohen's kappa coefficient assessed faculty interrater reliability and student-faculty reliability. Changes in scores between the two exercises were evaluated using Wilcoxon signed rank and paired t-tests. Spearman's correlation coefficient and repeated measures analysis assessed the association between shape carving scores and DAT, QR, and AA scores.
Results:
Faculty interrater reliability ranged from 0.2554 to 0.4186 for the assessed criteria over the two exercises. Student-faculty rater reliability ranged from 0.1299 to 0.4461. Only PAT scores correlated significantly to higher scores for the four grading criteria for both exercises. Student self-assessment improved with the second exercise.
Conclusions:
This rudimentary exercise was useful to identify students early in their dental education who may require more guidance in other manual dexterity courses. Faculties require additional training and calibration. Self-assessment is a life-long process of critical thinking that students are just beginning to learn.
Introduction
The quality of experiential education for pharmacists‐in‐training is heavily impacted by the habits of their preceptors. Assessment of preceptor capabilities and evidence of development has been limited due to the absence of a tool to meaningfully facilitate preceptor assessment in a robust and progressive fashion.
Objective
The primary objective was to determine the change in perceived precepting behavior and abilities of residents and preceptors enrolled in a teaching and learning curriculum (TLC) using a self‐directed assessment seeking (SDAS) approach via the Habits of Preceptors Rubric (HOP‐R).
Methods
Pharmacists enrolled in the 2018 to 2019 cohort of the Clinician Educators Program (CEP) TLC were invited to assess their precepting habits using the HOP‐R after attending the first seminar and again toward the end of the longitudinal 12‐month program using a SDAS approach. Using online surveys, participants rated their precepting capabilities as developing, proficient, accomplished, or master level for each of the 11 preceptor habits within the HOP‐R. The change in their self‐reported ratings was evaluated.
Results
The response rate for participants completing both the initial and follow‐up surveys was 56% (52/93). When comparing matched responses, the difference in self‐reported habit level ratings was statistically significant. The overwhelming majority of participants, 94%, reported advancement in one or more habits, with 60% reporting advancement in six or more habits while enrolled in the CEP.
Conclusion
Preceptor and resident pharmacists reported advancement of precepting capabilities within the conceptual framework of the HOP‐R while enrolled in a TLC. The SDAS approach yielded participant comments showcasing the perceived value of completing the criterion‐referenced HOP‐R assessment to foster self‐awareness and self‐efficacy related to precepting and preceptor development. The HOP‐R can be utilized within TLC to allow participants to assess precepting habits and show growth and development of their capabilities over time both individually and in aggregate.
To compare how well graduates of a self-directed, problem-based undergraduate curriculum (at McMaster University [MU], Hamilton, Ont.) and those of a traditional curriculum (at the University of Toronto [UT]) who go on to primary care careers keep up to date with current clinical practice guidelines.
Analytic survey. Management of hypertension was chosen as an appropriate topic to assess guideline adherence. An updated version of a previously validated questionnaire was mailed to the participants for self-completion.
Private primary care practices in southern Ontario.
A random sample of 48 MU graduates and 48 UT graduates, stratified for year of graduation (1974 to 1985) and sex, who were in family or general practice in Ontario; 87% of the eligible subjects in each group responded.
Overall and component-specific scores; analysis was blind to study group.
The overall mean scores were 56 (68%) of a possible 82 for the MU graduates and 51 (62%) for the UT graduates (difference between the means 5, 95% confidence interval 1.9 to 8.2; p < 0.01). Multivariate linear regression analysis of seven factors that might affect questionnaire scores revealed that only the medical school was statistically significant (p < 0.01). The MU graduates had significantly higher mean scores than the UT graduates for two components of the questionnaire: knowledge of recommended blood pressures for treatment (p < 0.05) and successful approaches to enhance compliance (p < 0.005). The trends were similar for the other components but but were not significant.
The graduates of a problem-based, self-directed undergraduate curriculum are more up to date in knowledge of the management of hypertension than graduates of a traditional curriculum.
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
The competent physician pursues lifelong learning through the recognition of deficiencies and the formulation of appropriate learning goals. Despite the accepted theoretical value of self-assessment, studies have consistently shown that the accuracy of self-assessment is poor. This paper examines the methodological issues that plague the measurement of self-assessment ability and presents several strategies that address these methodological problems within the current paradigm. In addition, the article proposes an alternative conceptualization of self-assessment and describes its associated methods. The conclusions of prior research in this domain must be re-examined in light of the common pitfalls encountered in the design of the studies and the analyses of the data. Future efforts to elucidate self-assessment phenomena need to consider the implications of this review.
People feel they understand complex phenomena with far greater precision, coherence, and depth than they really do; they are subject to an illusion-an illusion of explanatory depth. The illusion is far stronger for explanatory knowledge than many other kinds of knowledge, such as that for facts, procedures or narratives. The illusion for explanatory knowledge is most robust where the environment supports real-time explanations with visible mechanisms. We demonstrate the illusion of depth with explanatory knowledge in Studies 1-6. Then we show differences in overconfidence about knowledge across different knowledge domains in Studies 7-10. Finally, we explore the mechanisms behind the initial confidence and behind overconfidence in Studies 11 and 12. Implications for the roles of intuitive theories in models of concepts and cognition are discussed.
Although expected of all health professionals, self-assessment skills are seldom addressed directly in training. A previous review by the author identified curricular criteria associated with improved accuracy and validity in self-assessments of knowledge and performance in curriculum studies published between January 1970 and February 1990. The present review analyzed 11 studies that meet those criteria. Eight studies were of implementations of self-assessment components within training programs in the health professions, and three involved other training environments. Most described initial disorientation or opposition on the part of learners, attributed to unfamiliar roles and to learners' distrust. The curricula that successfully negotiated the transition to self-assessment norms reported noncognitive benefits such as improvements in morale, motivation, and communications among learners and faculty. Reported cognitive benefits included improvements in knowledge, performance, and self-analysis of performance. The constellation of effects suggests that effective self-assessment programs may promote more mature, collegial, and productive learning environments, particularly suited to the training of health professionals. Most curricula fostering effective self-assessment did not require extraordinary resources, and none jeopardized traditional standards. No evidence was found to support or challenge the expectation that self-assessment training would transfer to later work settings.
Valid self-assessment is fundamental to continuing professional competence but is seldom explicitly taught in health professions training. This review analyzed 18 scholarly articles published between January 1970 and February 1990 (14 articles regarding health professions trainees, and four concerning college students or graduate trainees) in which it was possible to compare performance as self-assessed by trainees with performance as assessed by experts or objective tests. The validity of self-assessed performance was found to be low to moderate and did not improve with time in conventional health professions training programs. Self-assessed performance seemed closely related to generalized self-attributions and was minimally influenced by external feedback in the form of test scores, grades, or faculty assessments. In five programs emphasizing explicit self-assessment goals and training strategies, moderate-to-high validity outcomes or improvements over time were demonstrated. Much of what passes for self-assessment in training seems the exercise of an underdeveloped skill, but effective training to improve validity and accuracy is available and feasible.
Medical students in a tutorial program rated themselves in Year 1 and again in Year 2 on Knowledge, Reasoning, Communication and Interpersonal Skills. Student and tutor ratings were similar in Year 1 but different in Year 2. Both self and tutor ratings increased from Year 1 to Year 2 but self-ratings were independent of achievement while tutor ratings were related to achievement. Possible reasons for these results were discussed with reference to the learning goals of the tutorials. © 1989 Informa UK Ltd All rights reserved: reproduction in whole or part not permitted.
The purpose of this paper was to determine if the decline in performance with time since completion of training on the 1980 ABIM Recertification Examination can be explained by a difference in performance on items testing different types of knowledge. Results showed that candidates further out of training performed less well on items testing new or changing knowledge, while performance on items testing stable knowledge was relatively constant across age groups.
Although self-evaluation is crucial in the practice of medicine, few educators have formally introduced self-assessment into the undergraduate medical curriculum. However, students in the baccalaureate-M.D. degree program at the University of Missouri, Kansas City, must complete a self-evaluation at the close of every medical school course and rotation during the last four years of a six-year curriculum. In this paper, the authors examine the self-ratings of 211 of these students as they progressed through the program in order to discover trends in and correlates of the self-assessments. Although the students' self-evaluations and faculty members' ratings of these students' performances rose year by year, the relationship between the students' and the faculty's ratings decreased through time. Yet, results suggest that self-evaluation has educational merit as a measure of noncognitive abilities associated with clinical performance and as a stimulus to further learning and professional development.
To determine whether continuing medical education affects the quality of clinical care, we randomly allocated 16 Ontario family physicians to receive or not receive continuing-education packages covering clinical problems commonly confronted in general practice. Over 4500 episodes of care, provided before and after study physicians received continuing education, were compared with preset clinical criteria and classified according to quality. Although objective tests confirmed that the study physicians learned from the packages, there was little effect on the overall quality of care. When the topics were of relatively great interest to the physicians, the control group (who did not receive the packages) showed as much improvement as did the study group. When the topics were not preferred, however, the documented quality of care provided by study physicians rose (P less than 0.05) and differed from that provided by control physicians (P = 0.01). Finally, there was no spillover effect on clinical problems not directly covered by the program. In view of the trend toward mandatory continuing education and the resources expended, it is time to reconsider whether it works.
As physicians, whether serving individual patients or populations, we always have sought to base our decisions and actions on the best possible evidence. The ascendancy of the randomized trial heralded a fundamental shift in the way that we establish the clinical bases for diagnosis, prognosis, and therapeutics. The ability to track down, critically appraise (for its validity and usefulness), and incorporate this rapidly growing body of evidence into one's clinical practice has been named 'evidence-based medicine' (EBM).
The goal of the study was to examine critically the use of taxonomic level as a criterion for determining the utility of a test item. If the distinction is useful, several hypotheses follow: There should be reasonable agreement among raters about taxonomic level; the taxonomic level should be independent of the level of the rater; in contrasting students from PEL and conventional curricula, since PEL students consistently report that they spend more time on conceptualizing and less on memorizing, there should be differential performance on factual recall and higher order items; higher order items should have greater predictive validity (addressed in other studies). Test items from the 1993 and 1994 Medical Council of Canada multiple choice examinations were reviewed by two recent graduates and two faculty. Although there was moderate agreement among raters, faculty were significantly more likely to rate items as Factual Recall. Performance of McMaster students and other Canadian graduates were then compared at an item level. No systematic differences between the two cohorts related to taxonomic level were identified. In conclusion, while there is moderate agreement about the taxonomic level of a test item, this appears to be related to educational level, and has little heuristic value as an indicator of performance.
The progress test (or Quarterly Profile Examination), invented concurrently by the University of Missouri-Kansas City School of Medicine and the University of Limburg, is used to provide useful summative and formative judgments about students' knowledge without distorting learning. All students in all classes sit the same examination at regular intervals through the year, and their individual progress is noted. This paper reports on four years' experience with a progress test, the Personal Progress Index (PPI), at the McMaster University Faculty of Health Sciences.
The PPI, a 180-item multiple-choice test with items drawn from all disciplines of medicine, is administered to medical students in all three classes three times per year. Individual feedback is provided, and accumulated student performance is determined for summative purposes. This paper examines extensive evidence of reliability, validity, and effect on student learning, using samples from the entering classes of 1992-1995 (a total of 400 students).
Reliabilities across test administrations (within classes) ranged from .46 to .63. The PPI demonstrated strong construct validity, with highly significant statistical tests of differences between classes and changes within classes on successive administrations. The predictive validity of the PPI, i.e., whether it could predict performance on the licensing examination of the Medical Council of Canada, increased monotonically from a correlation of .12 for the first test administration one month into medical school to a high of about .60 for the cumulative score across all administrations three months prior to the examination.
The PPI seems to be performing as intended, with students showing growth in performance across the three years of the MD program. Additional benefits are that many more students now self-refer for remediation (based on low PPI scores) and that the consistent relative performances of individual students across test administrations allow for the identification of students who have severe and persistent problems.
Although studies have examined medical students' ability to self-assess their performance, there are few longitudinal studies that document the stability of self-assessment accuracy over time. This study compares actual and estimated examination performance for three classes during their first 3 years of medical school.
Students assessed their performance on classroom examinations and objective structured clinical examination (OSCE) stations. Each self-assessment was then contrasted with their actual performance using idiographic (within-subject) methods to define three measures of self-assessment accuracy: bias (arithmetic differences of actual and estimated scores), deviation (absolute differences of actual and estimated scores), and covariation (correlation of actual and estimated scores). These measures were computed for four intervals over the course of 3 years. Multivariate analyses of variance and correlational analyses were used to evaluate the stability of these measures.
Self-assessment accuracy measures were relatively stable over the first 2 years of medical school with a decease occurring in the third year. However, the correlational analyses indicated that the stability of self-assessment accuracy was comparable to the stability of actual performance over this same period.
The apparent decline in accuracy in the third year may reflect the transition from familiar classroom-based examinations to the substantially different clinical examination tasks of the third year OSCE. However, the stability of self-assessment accuracy compares favorably with the stability of actual performance over this period. These results suggest that self-assessment accuracy is a relatively stable individual characteristic that may be influenced by task familiarity.
The ends, and beginnings, or science in the head: coming to grasp the causal structure of the world
- F Keil