No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
How Can I Know What I Don't Know? Poor Self Assessment in a Well-Defined Domain
Abstract
As the rapidity with which medical knowledge is generated and disseminated becomes amplified, an increasing emphasis has been placed on the need for physicians to develop the skills necessary for life-long learning. One such skill is the ability to evaluate one's own deficiencies. A ubiquitous finding in the study of self-assessment, however, is that self-ratings are poorly correlated with other performance measures. Still, many educators view the ability to recognize and communicate one's deficiencies as an important component of adult learning. As a result, two studies have been performed in an attempt to improve upon this status quo. First, we tried to re-define the limits within which self-assessments should be used, using Rosenblit and Keil's argument that calibration between perceived and actual performance will be better within taxonomies that are regularly tested (e.g., factual knowledge) compared to those that are not (e.g., conceptual knowledge). Second, we tried to norm reference individuals based on both the performance of their colleagues and their own historical performance on McMaster's Personal Progress Inventory (a multiple choice question test of medical knowledge). While it appears that students are able to (a) make macro-level self-assessments (i.e., to recognize that third year students typically outperform first year students), and (b) judge their performance relatively accurately after the fact, students were unable to predict the percentage of questions they would answer correctly with a testing procedure in which they have had a substantial amount of feedback. Previous test score was a much better predictor of current test performance than were individuals' expectations.
... Ehrlinger en el año 2008 16 demostraron que las autoevaluaciones de competencia en tareas específicas a menudo difieren significativamente de las evaluaciones objetivas de rendimiento. En el contexto educativo, estudios como el del Kevin Eva (2004) 17 han mostrado que los estudiantes frecuentemente sobreestiman su nivel de comprensión y habilidades, lo que subraya la necesidad de métodos de evaluación objetivos que se adhieran a los de autopercepción. ...
... Ehrlinger en el año 2008 16 demostraron que las autoevaluaciones de competencia en tareas específicas a menudo difieren significativamente de las evaluaciones objetivas de rendimiento. En el contexto educativo, estudios como el del Kevin Eva (2004) 17 han mostrado que los estudiantes frecuentemente sobreestiman su nivel de comprensión y habilidades, lo que subraya la necesidad de métodos de evaluación objetivos que se adhieran a los de autopercepción. ...
Objetivo: El objetivo de este estudio fue analizar las propiedades psicométricas de un cuestionario que permita analizar el conocimiento de docentes y directivos de establecimientos educacionales respecto a los beneficios de la actividad física sobre los procesos cognitivos de niños, niñas y adolescentes. Métodos: Se elaboró un cuestionario ad hoc, el cual se validó mediante un comité de 3 expertos utilizando el método Delphi junto a al índice de validez de contenido (IVC), y posteriormente un análisis de consistencia interna a través alfa de Cronbach. Resultados: El cuestionario quedo compuesto por un total de 27 ítems agrupados en 3 subescalas (Actividad física, Cognición y funciones ejecutivas y Actividad física y cognición en niños, niñas y adolescentes), y se aplicó a 64 profesores y directivos. Los resultados muestran un coeficiente α=0,960; ω=0.961 y un IVC = 0,87. Conclusión: Se considera que el instrumento puede ser un aporte relevante para generar estudios relativos a la temática de la relación entre la actividad física y la cognición, entregando información valiosa que permite profundizar los análisis en el marco del desarrollo de esta línea de investigación e implementación de acciones desde las políticas públicas nacionales.
... the only scoping review [20] conducted in medical education pointed out the misled emphasis on improving the accuracy of sa as the accuracy itself does not necessarily correlate with student learning; instead, research on sa should be guided towards sa for learning, which is, how learners achieve desired learning goals through the practice of understanding their current performance as compared to standards or rubrics for them to meet their goals. in fact, eva et al. [21] conducted a longitudinal study which revealed that medical students' accuracy in sa did not improve, even after 2.5 years of experience and feedback in sa. While not specifically focusing on sa, one scoping review of self-monitoring [22] suggested that self-monitoring aligns with more objective performance measures and is beneficial for improving performance. ...
... although the correlation between sa and other measures of academic performance has often been characterized as weak, sa remains a vital element in medical education, enabling students to acknowledge and articulate their learning gaps [21]. Future research could pivot from solely assessing the accuracy of sa in comparison to other evaluative measures. ...
Background
Self-assessment (SA) is increasingly recognized not only as an assessment method but also as a vital learning activity that enables learners to identify their strengths and limitations, fostering a crucial skill for lifelong learning. Despite its acknowledged significance, there remains a gap in understanding SA for and as learning, rather than merely of learning. This review, therefore, explores two primary questions: (1) What factors inform the implementation of medical students’ SA practices? (2) How does SA practices contribute to various educational outcomes?
Methods
A systematic review was conducted across five databases, employing a combination of keywords pertinent to SA and medical education. Our selection criteria deliberately excluded articles that solely focused on the accuracy of SA or used SA exclusively as a measure for other outcome variables. As a result, 39 studies met our inclusion criteria and were analyzed for this review. Qualitative narrative synthesis was used to summarize the informing factors. Furthermore, Kirkpatrick’s model was employed to categorize and summarize the effects of SA activities across various educational outcome levels, including reactions, learning, and behavioral changes.
Results
Our review reveals that key factors influencing students’ SA includes feedback from various stakeholders, peer assessment, the format of SA, and both learner and teacher training. Among the 39 studies examined, the majority demonstrated positive effects of SA on aspects such as attitudes and skills. However, a minority found no significant associations, with these outcomes frequently linked to contextual variables and how SA was implemented.
Conclusions
Our review does not focus on SA accuracy or its role as an assessment method; instead it delves into the role of SA as an integral learning practice. We explored its influencing factors and its impact across various outcomes. The findings indicate that for SA to effectively enhance student learning outcomes, it must be implemented with ample support, clear guidelines, and within contexts that encourage feedback and reflection.
... By analyzing the distribution of induced states, we found that the 'think-right' states account for a high proportion. This finding was consistent with overconfidence is common among learners [96,97]. Eva et al [97] found poor correlations between medical students' estimated knowledge and actual test scores. ...
... This finding was consistent with overconfidence is common among learners [96,97]. Eva et al [97] found poor correlations between medical students' estimated knowledge and actual test scores. They concluded that self-assessment is a poor predictor of actual performance. ...
Objective. Confusion is the primary epistemic emotion in the learning process, influencing students’ engagement and whether they become frustrated or bored. However, research on confusion in learning is still in its early stages, and there is a need to better understand how to recognize it and what electroencephalography (EEG) signals indicate its occurrence. The present work investigates confusion during reasoning learning using EEG, and aims to fill this gap with a multidisciplinary approach combining educational psychology, neuroscience and computer science. Approach. First, we design an experiment to actively and accurately induce confusion in reasoning. Second, we propose a subjective and objective joint labeling technique to address the label noise issue. Third, to confirm that the confused state can be distinguished from the non-confused state, we compare and analyze the mean band power of confused and unconfused states across five typical bands. Finally, we present an EEG database for confusion analysis, together with benchmark results from conventional (Naive Bayes, Support Vector Machine, Random Forest, and Artificial Neural Network) and end-to-end (Long Short Term Memory, Residual Network, and EEGNet) machine learning methods. Main results. Findings revealed: 1. Significant differences in the power of delta, theta, alpha, beta and lower gamma between confused and non-confused conditions; 2. A higher attentional and cognitive load when participants were confused; and 3. The Random Forest algorithm with time-domain features achieved a high accuracy/F1 score (88.06%/0.88 for the subject-dependent approach and 84.43%/0.84 for the subject-independent approach) in the binary classification of the confused and non-confused states. Significance. The study advances our understanding of confusion and provides practical insights for recognizing and analyzing it in the learning process. It extends existing theories on the differences between confused and non-confused states during learning and contributes to the cognitive-affective model. The research enables researchers, educators, and practitioners to monitor confusion, develop adaptive systems, and test recognition approaches.
... Furthermore, self-assessment is an important method in education, as it can be an effective learning tool. 8,9 Burrows 7 suggests that SA should be a collaborative process, with teachers and students developing reflective skills through mutual feedback. ...
... 4,5 Furthermore, effective feedback is also essential for effective self-assessment in improving student educational outcomes. 7,8 The SimEx dental education and evaluation system can provide students with accurate 3D real-time feedback, digital guidance and simulation, as well as objective computer-based assessment, facilitating student self-study and selfassessment and improving clinical skills. 10 These could be the reasons why for Q1 and Q4, all groups scored 4 or higher. ...
Background/purpose:
The SimEx is dental training system that applies new technology in a computerized dental simulator. The purpose of this study was to understand the usability satisfaction of the SimEx dental education and evaluation system by dental students and dentists at Tohoku University.
Materials and methods:
In this study, the Tohoku University IRB execution number was 2020-3-33. The number of subjects accepted was 59 at Tohoku University and divided into 4 groups based on years of clinical experience (Group A: 0 years; Group B: 1-2 years; Group C: 2-5 years; Group D: at least 5 years), and a total of 58 usability questionnaires were collected. Subjects completed the SimEx Usability Satisfaction Questionnaire after operating the SimEx (EPED Inc., Kaohsiung, Taiwan) course, which contained 16 questions.
Results:
Among the 58 questionnaires collected by Tohoku University, there were 19 undergraduate students (4th∼6th grade), 12 post-graduate students, 14 residents, and 13 dentists. Significant differences between Group A and Group B, and between Group A and Group D were found (P < 0.05). The same results were obtained for the "experience satisfaction index". In the items where significant differences were found, longer clinical experience tended to result in lower scores.
Conclusion:
From these results, we can conclude that the SimEx education and evaluation system facilitates students' self-learning, and this system is very useful for continued study and clinical skill training for dentists, especially for students and junior dentists with high usability satisfaction.
... Furthermore, since the subject of the study is the perception of the owner-manager, who is generally the sole worker in the firm (or on average, has just one further employee), it is not possible to survey multiple respondents per firm. Face validity and subjective assessment validity (Hair, 1998;Hair et al., 2010) were satisfied through the use of existing methods of measurement where possible, such as Stokols et al. (2001) and Eva et al. (2004) for Perceived-Knowledge; and the pilot study (which ensured the understanding of questions and answers matched our expectations, Rong and Wilkinson, 2011). In addition, the survey was structured specifically to reduce the risk of respondents linking different concepts. ...
... Cronbach's alpha = 0.79. This variable is based upon prior literature including Stokols et al. (2001) and Eva et al. (2004), who both employ similar measures to test self-assessments of knowledge. Perceived-knowledge is used here, rather than a test of actual-knowledge because opinions of value, burden, and compliance level are based on what owner-managers think they know, rather than their actual knowledge, which is likely different ...
Business regulation is only successful when firms comply. Compliance is determined by several factors, including knowledge-of/attitude-towards regulation, which we suggest consists of both burden and value. Existing literature generally ignores regulatory value and focuses on larger firms in “high-risk” industries. We explore Perceived Own-Firm and Perceived Competitor Compliance in English accommodation micro-firms. Perceived Own-Firm Compliance exceeds Perceived-Knowledge of regulation, and has the strongest (positive) association with Perceived-Value. Perceived Competitor Compliance is thought to be very low and is (negatively) associated with Perceived-Value and Perceived-Burden. Furthermore, the factors associated with greater Perceived Own-Firm Compliance are associated with lower Perceived Competitor Compliance.
... Sense of Urgency Vs Actual Urgency Students described feeling a sense of urgency to treat the patient quicker, however this did not align with their actual time to treat and decision making patterns.This suggests that participants' perceptions of performance may not align with actual performance. This phenomenon is not new -Eva et al (2004) demonstrate the correlation between self-assessment and actual performance in students is poor[194] -"How can I know what I don't know?". In a practical sense, authentic portrayal of moulage might provide more opportunities for students to bridge the gap between self-awareness and performance; moulage presents an opportunity to deliver feedback from objective ...
... Sense of Urgency Vs Actual Urgency Students described feeling a sense of urgency to treat the patient quicker, however this did not align with their actual time to treat and decision making patterns.This suggests that participants' perceptions of performance may not align with actual performance. This phenomenon is not new -Eva et al (2004) demonstrate the correlation between self-assessment and actual performance in students is poor[194] -"How can I know what I don't know?". In a practical sense, authentic portrayal of moulage might provide more opportunities for students to bridge the gap between self-awareness and performance; moulage presents an opportunity to deliver feedback from objective ...
Moulage in the traditional sense is the art of replicating illnesses and wounds through casting wax moulds. Origins are traced to Ancient Egypt and forbidden practices of 17th century Europe. While traditional moulage is now housed in musea across the world, modern moulage is used to replicate illness and effects in simulation using special effects makeup techniques. Simulation is a well-established technique to prepare health professionals for clinical practice, and is grounded in a strong evidence base. Despite the strong evidence for the use of simulation, the conditions of moulage is an underexplored topic within the context of simulation research, and we know very little regarding how it works, under what conditions and what the effect is on participants of simulation. In order to better understand how and why moulage impacts on participants of simulation, a series of complementary studies were completed. Initially a Systematic Review of authentic moulage in simulation was undertaken to understand the current research on moulage. This provided a useful baseline for the current use and evidence for moulage in simulation. Subsequently, a further study was undertaken to define authentic moulage in simulation via an electronic Delphi consensus method. This study recruited international experts on moulage and resulted in the development of the Moulage Authenticity Rating Scale (MARS) to measure moulage authenticity. Finally, a third study was conducted to explore how the authenticity of moulage effects participant engagement in simulation using a randomized control experiment design. This study utilized the MARS tool developed from the previous study to design moulage that was low-authenticity and high-authenticity, and compared levels of engagement using measures of self-report, eye tracking and interview methods. The results of this work presents previously unrecognized information on how medical students perceive the authenticity of moulage and how it contributes to their performance and engagement in simulation. In summary, I present a number of suggestions as to how simulation users and designers might consider moulage in their everyday practice. This thesis presents a series of philosophical research questions and findings that collectively make an original contribution to the future of moulage in simulation and undergraduate Medical Education using simulation-based curriculum, teaching and learning.
http://hdl.handle.net/1959.13/1412570
... Self-assessment of learning needs via reflection on action has been widely recognized as a valuable form of situated learning [72]. Unfortunately, the validity of self-assessment is moderate at best, in part due to difficulty perceiving one's own deficiencies and limitations in identifying one's own tacit knowledge or recalling how one has learned in the past, either formally or informally [73][74][75]. Furthermore, personal characteristics, emotional responses, and interpersonal dynamics impact self-assessment in complex ways not yet fully understood [76]. ...
Background
Like other countries developing standardized general practice training, China faces the challenge of training vast numbers of new general practice faculty. However, little is known about these clinician-teachers’ motivations and perceived needs for faculty development. This review intended to explore available published data on Chinese general practice faculty development needs and motivation for ongoing professional development.
Methods
A systematic search was conducted using combinations of keyword and MeSH term searches in English in PubMed and in Chinese language CNKI and WanFang databases, followed by a qualitative narrative synthesis of relevant articles identified. Inclusion criteria: English or Chinese publications between 1990 and 2021; qualitative or quantitative research, systematic reviews, literature reviews, review or opinion articles pertaining to faculty development or professional development needs or motivations of general practice or family medicine teachers. Articles pertaining to fields outside of medicine, those with a heavy subspecialty focus or not relevant to primary care were excluded.
Results
A total of 88 full text articles were included in this review. Available data on Chinese general practice faculty development needs placed emphasis on fundamental general practice knowledge, principles, and clinical training needs more than teaching and assessment skills. Learning through teaching and a sense of responsibility were potential sources of intrinsic motivation for pursuing faculty development. Potential barriers to participation included busy workloads, inadequate organizational support, and limited finances. Mandatory participation, irrelevant content, lack of teaching experience, lack of motivation to teach, and overall job dissatisfaction appeared to negatively influence motivation to pursue faculty development.
Conclusions
Further research should explore the broader continuous professional development needs of this large population of general practice teachers. Qualitative studies are needed to clarify how individual, organizational, and contextual factors influence teachers’ motivation to pursue faculty development.
... Usage rates indicate students valued FMCQs as a learning tool and to aid self-assessment, which likely supplements other learning methods such as Anki (which uses spaced repetition to increase retention). 10 We did not examine if PREP usage impacted final summative examination performance here, but plan to do so in the future. Other groups have shown that student-generated FMCQs can be representative of faculty-written Table 2 PREP a Percentage who agree was defined as the percentage of respondents who selected "agree" or "strongly agree." ...
Problem
A shortage of curriculum-aligned formative multiple-choice questions (FMCQs) remains despite their known learning benefits in preclinical medical education due to limitations on teaching faculty time and other reasons. In response, students often use extramural resources such as commercial or collaborative question banks; however, these options are often expensive and cannot be aligned with the content of each school’s unique curriculum. In addition, students need feedback on their learning in a manner that parallels the format of summative assessments. In this pilot, the authors aimed to enhance student learning by creating an intramural formative practice resource that was developed as the curriculum unfolded under the direction of the faculty leading the concurrently running curricular units.
Approach
The authors developed a workflow known as Professor-Reviewed Exam Practice (PREP) in 2023. PREP partnered preclinical medical students and faculty to create vignette-style, single best response FMCQs with feedback for every lecture and self-guided learning module in multiple preclinical blocks of The Ohio State University College of Medicine undergraduate medical curriculum.
Outcomes
PREP established a sustainable, student-led, faculty-guided workflow that created high-quality, curriculum-aligned FMCQs for student use in the preclinical medical curriculum over a 14-month period. Usage rates were high across multiple preclinical blocks, reflecting high student demand for FMCQs of this nature and their value as a study aid. Survey data showed faculty agreed that their time commitment and role in the PREP workflow was appropriate.
Next Steps
Future work will evaluate the benefits of PREP to students by exploring the potential impact of PREP FMCQs on summative assessment performance and if writing FMCQs confers benefits to PREP team members. Faculty survey indicated that performance data from PREP FMCQs could be used to tailor upcoming teaching and learning methods, which is an area for future inquiry.
... Self-determination theory acknowledges the independent natural motivations of individuals to learn and improve, but also recognizes the important role of coaches as a source for valuable change-oriented feedback [27][28][29][30]. Previous studies have shown that trainees often need support in performing self-assessment and that unguided self-assessment is often flawed as trainees most often tend to overestimate their performance [31][32][33]. In line with the findings of this study, others have also reported the importance of coaches being able to unpack and discuss disconfirming feedback within the coaching relationship. ...
Purpose
We implemented a near-peer (resident–student) coaching intervention in 2021–2022 to improve clerkship students’ health system science (HSS) learning and application. This study thus sought to identify implementation barriers and potential facilitators of an effective HSS near-peer coaching program for clerkship students from the resident coach’s perspective at a single tertiary-care institution.
Methods
A mixed-methods study design was employed. Thirty surgical residents volunteered to serve as HSS coaches for rotating third-year medical students during the 2021–2022 academic year. Coaches were asked to complete an exit survey, and a subset was purposefully sampled to complete semi-structured interviews. Transcripts were coded and analyzed with a framework method to identify emergent themes indicative of the barriers and facilitators of effective implementation of this near-peer HSS coaching program.
Results
A total of 20/30 (67%) resident coaches completed the survey and 50% of these residents ( n = 10/20) completed an interview. Coaches facilitated multiple meetings (ranging from 1 to more than 5 times) with their assigned students over the course of the 3-month surgical rotations. Most coaches who deemed themselves successful in coaching reported early prioritization of setting individualized clerkship goals. Rapport building and strong communication were also important factors identified in facilitating effective near-peer cognitive coaching relationships.
Conclusions
Our study suggests that, from residents’ perspective, effective near-peer coaching by residents on surgical clerkships requires special emphasis on early goal setting, as well as training and development of coaches in skills, such as feedback delivery, and rapport building to encourage strong resident–student coaching relationships that cater to the individualized needs of the learner.
... In healthcare, physicians' self-assessment capabilities and subsequent self-regulated learning efforts are closely linked to the quality of care they provide (Murdoch-Eaton & Whittle, 2012;Sandars & Cleary, 2011). However, the evidence demonstrates a weak correlation between physicians' self-rated assessments and standardized external assessments (Davis et al., 2006;Eva et al., 2004). Among health profession trainees, the correlation between self and external assessments of knowledge has been shown to range between 0.02 and 0.65 (Gordon, 1991). ...
Studying texts constitutes a significant part of student learning in health professions education. Key to learning from text is the ability to effectively monitor one’s own cognitive performance and take appropriate regulatory steps for improvement. Inferential cues generated during a learning experience typically guide this monitoring process. It has been shown that interventions to assist learners in using comprehension cues improve their monitoring accuracy. One such intervention is having learners to complete a diagram. Little is known, however, about how learners use cues to shape their monitoring judgments. In addition, previous research has not examined the difference in cue use between categories of learners, such as good and poor monitors. This study explored the types and patterns of cues used by participants after being subjected to a diagram completion task prior to their prediction of performance (PoP). Participants’ thought processes were studied by means of a think-aloud method during diagram completion and the subsequent PoP. Results suggest that relying on comprehension-specific cues may lead to a better PoP. Poor monitors relied on multiple cue types and failed to use available cues appropriately. They gave more incorrect responses and made commission errors in the diagram, which likely led to their overconfidence. Good monitors, on the other hand, utilized cues that are predictive of learning from the diagram completion task and seemed to have relied on comprehension cues for their PoP. However, they tended to be cautious in their judgement, which probably made them underestimate themselves. These observations contribute to the current understanding of the use and effectiveness of diagram completion as a cue-prompt intervention and provide direction for future research in enhancing monitoring accuracy.
... Usage rates indicate students valued FMCQs as a learning tool and to aid self-assessment, which likely supplements other learning methods such as Anki (which uses spaced repetition to increase retention). 10 We did not examine if PREP usage impacted final summative examination performance here, but plan to do so in the future. Other groups have shown that student-generated FMCQs can be representative of faculty-written Table 2 PREP a Percentage who agree was defined as the percentage of respondents who selected "agree" or "strongly agree." ...
... Dudo (2013) also found that participants who had higher communication self-efficacy felt empowered to engage in more public science events than those who have not received training. However, many trainers and researchers have also identified several limitations to this form of self-report measurement, such as trainees inflating their self-assessed score as well as the desire to provide positive feedback, even if skills were not developed (Eva et al., 2004;Mort & Hansen, 2010). Recent studies have thus begun to consider the importance of external evaluation of communication skill by experts, peers, or target audience members (David & Baram-Tsabari, 2019). ...
This study conducted a systematic quantitative literature review to examine the characteristics of communication training programs for Science, Technology, Engineering, and Mathematics (STEM) professionals and their overall impacts related to improving participants’ oral communication skills. Sixteen studies were identified and included in the analysis. Results demonstrate an overall positive effect of communication training programs on the oral communication skills of STEM professionals and highlight the need for integrating theoretical frameworks, using multiple methodological approaches, and incorporating validated measures into the evaluation of training programs. These findings provide meaningful insights for future research and practice in the training and development subfield of STEM communication.
... Они могут более осознанно контролировать свой учебный процесс, а также свои мыслительные и поведенческие привычки. [15] Критическое мышление также пригодится учителям при решении сложных задач, которые могут возникнуть в учебной среде. ...
Статья анализирует и сравнивает традиционную и пересмотренную таксономию Блума, фокусируясь на когнитивном домене и, в частности, на знании. Предлагается к рассмотрению двумерная модель пересмотренной таксономии Блума, так как сейчас в казахстанском образовании сложилась интересная ситуация – иерархия когнитивных навыков используется из классической таксономии, при этом концепция активных глаголов применяется уже из пересмотренной. Мы считаем, что необходимо в таком случае полностью использовать пересмотренный концепт, и в статье приводится пример матрицы для формулирования образовательных целей. Особое внимание уделяется когнитивному навыку применения, который некоторые классифицируют как низший уровень мышления, в то время как другие считают его высшим. В статье высказывается мнение, что этот навык является переходным, перекликаясь с обоими уровнями. Кроме того, в статье исследуется взаимосвязь между критическим мышлением и таксономией Блума в контексте подготовки учителей. Рассматривается, как использование таксономии Блума может способствовать развитию критического мышления у будущих педагогов, какие преимущества это может иметь в образовательной практике. Статья позволяет лучше понять эволюцию таксономии и описания когнитивных навыков и способностей, представленных в таксономии Блума, их роль в формировании компетентных учителей с умением критически мыслить и эффективно применять знания в учебной среде
... Such contradiction may put forward the idea of leaving the responsibility of assessment to learners. However, such ideas have proven to be too controversial (Deci & Ryan, 2008;Eva et al., 2004). It is also argued that even though PBL is conducive to students' effective learning, it seems challenging to assess learners' performance during the course (Shamsan & Syed, 2009). ...
Problem-based learning (PBL), as one of the trendy learning and teaching approaches, has been investigated in various disciplines. Following the cresting benefit and reputation of this methodology in medicine (Vleuten & Schuwirth, 2019), it soon grabbed scholars' attention in the field of language teaching. The present exploratory mixed-method study was an attempt to identify the potentially impeding factors that prevent language teachers from offering problem-based teaching in their classes despite its reported advantages. To this end, in the qualitative phase, 20 EFL teachers took part in a semi-structured interview. The interviews were transcribed, coded, categorized, and thematized. In the quantitative phase, a 66-item Likert scale questionnaire was developed based on the qualitative phase results and was administered to 200 EFL teachers. The results indicated that factors such as difficulties in designing a PBL problem/scenario, the role of teaching equipment, teachers' unfamiliarity with designing a problem/scenario, limiting PBL to a specific skill, assessment, students/teachers' unfamiliarity with the PBL, students' cross-cultural differences, and limiting PBL to a specific level of language proficiency discourage teachers from using this approach. The implications of the findings are discussed and suggestions are made for future research.
... Such contradiction may put forward the idea of leaving the responsibility of assessment to learners. However, such ideas have proven to be too controversial (Deci & Ryan, 2008;Eva et al., 2004). It is also argued that even though PBL is conducive to students' effective learning, it seems challenging to assess learners' performance during the course (Shamsan & Syed, 2009). ...
Problem-based learning (PBL), as one of the trendy learning and teaching approaches, has been investigated in various disciplines. Following the cresting benefit and reputation of this methodology in medicine (Vleuten & Schuwirth, 2019), it soon grabbed scholars' attention in the field of language teaching. The present exploratory mixed-method study was an attempt to identify the potentially impeding factors that prevent language teachers from offering problem-based teaching in their classes despite its reported advantages. To this end, in the qualitative phase, 20 EFL teachers took part in a semi-structured interview. The interviews were transcribed, coded, categorized, and thematized. In the quantitative phase, a 66-item Likert scale questionnaire was developed based on the qualitative phase results and was administered to 200 EFL teachers. The results indicated that factors such as difficulties in designing a PBL problem/scenario, the role of teaching equipment, teachers' unfamiliarity with designing a problem/scenario, limiting PBL to a specific skill, assessment, students/teachers' unfamiliarity with the PBL, students' cross-cultural differences, and limiting PBL to a specific level of language proficiency discourage teachers from using this approach. The implications of the findings are discussed and suggestions are made for future research.
... Several theories, including cognitive load theory, encourage maximum coaching early in the learning process [11,12]. Findings also suggest that learners struggle to self-assess their learning [13] and thus may make poor use of independent practice opportunities. Consistent with this, a recent study found that expert feedback during early deliberate practice supported greater learning of endourologic skills than did feedback provided during a later deliberate practice session [14]. ...
Background
Opportunities to practice procedural skills in the clinical learning environment are decreasing, and faculty time to coach skills is limited, even in simulation-based training. Self-directed learning with hands-on practice early in a procedural skill course might help maximize the benefit of later faculty coaching and clinical experience. However, it may also lead to well-learned errors if learners lack critical guidance. The present study sought to investigate the effects of a hands-on, self-directed “study hall” for central line insertion among first-year residents.
Methods
Learner cohorts before vs. after introduction of the study hall ( n = 49) were compared on their pre- and post-test performance of key procedural behaviors that were comparable across cohorts, with all learners receiving traditional instructor-led training between tests.
Results
Study hall participants spent a median of 116 min in hands-on practice (range 57–175). They scored higher at pre-test (44% vs. 27%, p = .00; Cohen’s d = 0.95) and at post-test (80% vs. 72%, p = .02; Cohen’s d = 0.69). A dose–response relationship was found, such that 2 h of study hall were roughly equivalent to the performance improvement seen with four clinical observations or supervised insertions of central lines.
Conclusions
Self-directed, hands-on “study hall” supported improved procedural skill learning in the context of limited faculty availability. Potential additional benefits make the approach worth further experimentation and evaluation.
... One weakness of all three of these inventories is that they are learning tools rather than formal tests of skill sets. In addition, there is a long-standing discussion across a wide range of disciplines about the accuracy of self-assessments when compared to observed measures of competence or objective tests (Falchikov & Boud, 1989;Wilson, 1999;Kruger & Dunning, 1999;Eva et al., 2004;Dunning et al., 2004;Davis et al., 2006). For example, in a meta-analysis of quantitative self-assessment studies in law, engineering, guidance counseling, behavioral science, psychology and medicine, Falchikov and Boud found correlations between self and external assessments of student performance ranging from .05 to .82, with a mean of .39. ...
This study investigates students' self-perceptions of learning related to their participation in either a 3- or 15-week skill-based graduate coaching course. The results indicate students enrolled in the compressed course saw a significant increase in their coaching skills from pre-test to post-test; students enrolled in the full-term did not see the same statistically significant increase. Further, students in the compressed course had statistically significant higher counseling and challenging skills than the full-term students. These findings are relevant to the debates surrounding the benefits accrued from obtaining an MBA. (original abstract)
... 24 Another advantage of CBL is that it promotes selfdirected learning (SDL), which is an important component of lifelong learning and therefore a vital competency in medical school curricula. 25 Various studies have described the use of CBL in anatomy education through graded oral presentations designed by the first-year medical students, 26 as well as the positive perception and efficacy of CBL in anatomy education compared to other teaching strategies. [27][28][29] One important but commonly overlooked aspect is the role of structured reviews in active learning. ...
Introduction
Recent changes in undergraduate medical curricula have resulted in time constraints that are particularly challenging, especially when students must learn large blocks of musculoskeletal anatomy content. Consequently, students have resorted to rote memorization to cope, which counteracts our established collaborative self-directed learning model.
Methods
For a 6-week musculoskeletal anatomy course, two structured case-based review sessions are described, each following the completion of two five-hour lab sessions, two on the upper extremities and two on the lower extremities. These largely self-directed review sessions consisted of 6 students rotating through 7 to 8 stations every 10 minutes where clinical cases with follow-up questions were projected on large screens. The students were expected to work collaboratively to solve the cases utilizing the prosected specimens provided and discuss the accompanying answers at the end of each case.
Results
Ninety-four per cent of the students who participated in this study agreed that the case-based review sessions provided a helpful overview of musculoskeletal anatomy content. Student performance on the open-ended, case-based musculoskeletal examination questions showed no significant difference in performance on shoulder, hand, hip, thigh, and leg questions. There was, however, a statistically significant decrease in the students’ scores on a forearm question in 2021 compared to 2019.
Conclusion
This paper describes our integrated, collaborative musculoskeletal course, including case-based review sessions, which was positively received by students as having value in reviewing the musculoskeletal content though it was not found to improve examination performance.
... Red herrings can be added to create garden path scenarios, thus allowing learners to discriminate between similar ailments, and recognize the need for supplemental information through labs or other means to produce a differential diagnosis. This may help learners to become cognizant of what they do not know which is fundamentally challenging (Eva et al., 2004). AR can aid in allowing learners to recognize gaps in their information or knowledge, and then relevant and appropriate means to fill those gaps can be utilized. ...
It is necessary to educate medical students to prepare them for the healthcare world they will enter upon completion of their training. This pertains to not only the content of the education, but also the experiences during the instruction, and the application of the learning. However, during their time in training, a student will likely not encounter all the ailments that they will while working. This means that their education should be supplemented to help them be better prepared. We propose augmented reality (AR) virtual patients as a method to assist in the delivery of this supplementation. This paper’s focus is on enhancing the usefulness of AR by providing learners with interactions with a variety of virtual patients. The target audience for this topic includes medical educators, medical school students, residents, educators interested in AR, and designers of educational AR systems. The mentioned topics can even apply to continuing education or refresher courses for physicians. A review of aspects that can allow AR to be more immersive and beneficial to those learning about various ailments is provided. Elements of AR for medical education training are discussed to enhance the relevance and applicability of the learning experience. These include presenting multiple simultaneous ailments and the ability to modify AR patient characteristics, which can be accomplished more quickly and with more possibilities than when using manikins.
... Ahol a kritikák "fogást találnak" az eredeti koncepción, az leginkább a pszichológiai, illetve statisztikai megalapozás, illetve az interpretáció (Christopher et al. 2021, Gignac 2022, Mcintosh et al. 2019, Pennycook et al. 2017, Zajenkowski et al. 2020. A kutatások egy másik, jelentős része arra irányul, hogy az egyes tevékenységek vagy képességek közti különbséget vizsgálja a DKH szempontjából (Dunning 2011, Ehrlinger et al. 2008, Eva et al. 2004, Nierenberg & Dahl 2021, vagyis azt, hogy eltérő kihívások esetén eltérően jelentkezik-e (ha jelentkezik egyáltalán) a hatás. Arra mindenképpen rávilágítanak ezek a kritikák, hogy a felszíni jelenség mögött álló mechanizmus, illetve a hatást befolyásoló tényezők feltárása egyelőre befejezetlen, és több, változatos területekről származó empirikus bizonyítékot igényel. ...
A TANULMÁNY CÉLJA Dunning–Kruger-hatásként (DKH) ismert az a jelenség, hogy a rosszabbul teljesítő egyének hajlamosabbak pontatlanabbul és felfelé torzítva becsülni saját teljesítményüket. Jelen kutatás e hatást kívánja mérni marketing kurzusok keretében a magyar felsőoktatásban, amivel bővíti a hazai szakirodalomban eddig, ilyen szempontból elemzett tantárgyak körét. ALKALMAZOTT MÓDSZERTAN Az adatgyűjtés 2021 májusában zajlott 3 alapképzéses szak marketing témájú kurzusain, online zárthelyi dolgozatok keretében (N=160). A hallgatók a dolgozat első és utolsó kérdéseként megbecsülhették az elért pontszámukat. A hatás definíciója alapján feltételeztük, hogy a magasabb pontszámot elérő hallgatók pontosabbak és kevésbé becsülik felül pontszámaikat a dolgozat megírása előtt és után egyaránt. További hipotézisünk szerint a jobb teljesítményűek több információt szereznek saját tudásukról a teszt alatt, ezért náluk a vizsga után átlagosan kisebb a pontszámok felülbecslése és a becslési hiba, mint előtte. Vizsgáltuk még, hogy a DKH egyformán jelenik-e meg a nőknél és a férfiaknál is. A kutatásban elsősorban többváltozós lineáris regressziós modelleket alkalmaztunk. LEGFONTOSABB EREDMÉNYEK Eredményeink alátámasztották, hogy minél gyengébb teljesítményt ért el egy hallgató, annál pontatlanabbul ítélte meg és egyben jobban felül is becsülte várható pontszámát. A jobban teljesítők becslései hajlamosabbak voltak a dolgozat végére optimistábbá és pontatlanabbá válni. Ugyanakkor a nemek nem mutattak összefüggést a vizsgált hatással. GYAKORLATI JAVASLATOK A kutatás eredményei alapján a gyengébben teljesítő hallgatók kevésbé vannak tisztában saját tudásukkal, így nem is érzik annyira szükségét a fokozottabb felkészülésnek. Ez az oktatók számára azt jelenti, hogy az oktatás során az ismeretátadáson túl e hatás csökkentésére, kiküszöbölésére is figyelmet kell fordítani, például a kijavított dolgozatok közös áttekintése révén, különösen, mivel a képzés során szerzett kudarcok a lemorzsolódás egyik okát jelentik.
... Ahol a kritikák "fogást találnak" az eredeti koncepción, az leginkább a pszichológiai, illetve statisztikai megalapozás, illetve az interpretáció (Christopher et al. 2021, Gignac 2022, Mcintosh et al. 2019, Pennycook et al. 2017, Zajenkowski et al. 2020. A kutatások egy másik, jelentős része arra irányul, hogy az egyes tevékenységek vagy képességek közti különbséget vizsgálja a DKH szempontjából (Dunning 2011, Ehrlinger et al. 2008, Eva et al. 2004, Nierenberg & Dahl 2021, vagyis azt, hogy eltérő kihívások esetén eltérően jelentkezik-e (ha jelentkezik egyáltalán) a hatás. Arra mindenképpen rávilágítanak ezek a kritikák, hogy a felszíni jelenség mögött álló mechanizmus, illetve a hatást befolyásoló tényezők feltárása egyelőre befejezetlen, és több, változatos területekről származó empirikus bizonyítékot igényel. ...
A TANULMÁNY CÉLJA Az önvezető járművek (AV) bevezetése forradalmasíthatja a mobilitást. Elterjedésük egyik legmeghatározóbb korlátja azonban társadalmi elfogadásuk. Az utóbbi években egyre nő azon szakcikkek száma melyek az AV-k fogyasztói elfogadását vizsgálják valamely technológiaelfogadási modell felhasználásával, kérdőíves megkérdezéssel. Mindinkább előtérbe kerül ugyanakkor ezen kutatási módszertanok azon releváns korlátja, mely szerint a válaszadók nem rendelkeznek személyes tapasztalattal az AV-ben történő utazást illetően. Tanulmányunk célja egy olyan kutatási módszertan fejlesztése és tesztelése, mely ötvözi a technológiaelfogadás és-használat egyesített elméletét (UTAUT) a conjoint elemzéssel. Célunk, hogy megállapíthassuk, melyek azok a legfontosabb termékjellemzők az AV-k tekintetében, amelyek leginkább befolyásolják a fogyasztókat döntéshozás közben. ALKALMAZOTT MÓDSZERTAN Új termékek fejlesztése során sikeresen alkalmazható módszertan a conjoint elemzés. Kutatásunk során ezért a korábban már validált UTAUT modell független változóit, mint termékjellemzőket azonosítottuk egy teljes profilú conjoint elemzés létrehozására. Primer adatgyűjtésünk során a conjoint kártyákat összesen 202 fő értékelte két lépésben. Elsőként saját preferenciáik szerint sorbarendezték, másodikként tíz pontos Likert skálán értékelték a kapott kártyákat, melyek a termékjellemzők szintjeinek különböző kombinációit jelenítették meg. LEGFONTOSABB EREDMÉNYEK Eredményeinkből kiderül, hogy az UTAUT modell segítségével létrehozott teljes profilú conjoint elemzés sikeresen alkalmazható az önvezető járművekhez kapcsolódó fogyasztói preferenciák vizsgálatára. Fontos ugyanakkor megjegyezni, hogy a Likert skála segítségével történő értékelés során megbízhatóbb eredményeket kaptunk, mint a preferenciasorrend felállítása esetén. Legfontosabb eredményünk, hogy válaszadóink számára a biztonság kiemelkedően a legfontosabb termékjellemző, melyet a funkcionális attribútumok némileg lemaradva, mégis kéz a kézben követnek, legkevésbé fontosak pedig a belső jellemzők. GYAKORLATI JAVASLATOK Eredményeinkkel szeretnénk felhívni a fejlesztők és döntéshozók figyelmét első sorban arra, hogy munkájuk során kiemelkedő helyet kapjon az AV-k biztonsága a széleskörű elterjedés érdekében. Irányadó lehet továbbá, hogy eredményeink alapján egy jól működő és kényelmes AV-ért a fogyasztók hajlandóak lennének magasabb árat fizetni. Köszönetnyilvánítás: Készült a Nemzeti Kutatási, Fejlesztési és Innovációs Hivatal – NKFIH K 137571 azonosító számú K_21 „OTKA" Kutatási témapályázat támogatásával.
... D 'Eon, Sadownik, Harrison, and Nation (2008) suggested that the aggregate retrospective post-then data may be a more useful measure to evaluate the overall impact of a teacher education program rather than to measure impact on individual participants. This approach may be useful for IHEs in lieu of the traditional pre/posttest assessments, particularly where participants may be likely to rate themselves higher on the pre-than the post-test, or provide socially desirable responses (Eva, Cunnington, Reiter, Keane, & Norman, 2004;Lam & Bengo, 2003). Faculty may also consider the retrospective post-then approach as a methodology for evaluating their overall teacher education program. ...
Professional development that focuses on supporting teachers' abilities to work with diverse families is keenly needed. This article outlines three instructional strategies and how they were used with undergraduate students in an inclusive early childhood teacher education program: (a) involving diverse family members as instructional supports; (b) performance of family scenarios; and (c) examining diverse families in popular media. These strategies, undergirded by the importance of critical reflection, have been implemented as part of a statewide preservice teacher education initiative in Iowa. Program evaluation data provide preliminary evidence of their potential to foster the development of necessary cultural dispositions for future teachers to meet the growing diversity among children and families.
... Current research has attempted to establish the correlation of SA accuracy with different factors. For example, both Ross (2006) and Eva, Cunnington, Reiter, Keane, & Norman (2004) mentioned that under-achieving students tend to overestimate their performance. Brantmeier, Vanderplank, & Strube (2012) investigated students' SA accuracy and their proficiency level and found that language proficiency level was positively correlated with SA accuracy. ...
Checklists, as a form of self-assessment, have been provided to students by educators as a way to promote learner autonomy. However, due attention has not been paid to the accuracy of those completed checklists. Through a survey, interviews, and comparison of students’ and researchers’ completed checklists, the research intends to investigate the accuracy of students’ self-assessment of their writing in a Sino-foreign university in China. Suggestions on checklist design and pedagogical implications on how to improve learners’ self-assessment are made.
... Reflective practice is an integral component of experiential and self-directed forms of learning (10). Research also indicates that reflective practices are significant predictors of SDL competencies (11) that are considered an important component of life-long learning and thus, a key competency in medical school curricula (12). Based on the principles of SDL, reflective practice motivates students to do self-assessment, improves critical thinking skills, encourages professionalism and even communication skills (13). ...
E-portfolios based on reflections and reflective practice enhance self-directed learning. The key components of self-directed learning according to Garrison's model include self-management, self-monitoring, and motivation. The aim of the present study was to explore students' perceptions of utilising learning portfolios as a tool for reflective practice and to evaluate their responses based on Garrison's Model of Self-Directed Learning. The current study was conducted among the second-year pre-clinical students at the College of Medicine and Health Sciences, National University of Science and Technology, Sultanate of Oman. 165 students, enrolled for the Behavioural Science in Medicine course were part of the study. They were given an orientation to reflective practice and portfolio writing. Feedback from students was obtained on completion of the course e-portfolio. A majority of students indicated that completing the e-portfolio enhanced self-directed learning, encouraged self-reflection, improved insight, and contributed towards self-awareness. Participants' responses based on Garrison's Model of Self-Directed Learning indicate above average mean scores across all three domains-self-management, self-monitoring, and motivation. Significant inter-domain correlation was also seen The e-portfolio is an effective modality to translate reflective practice to practical learning in undergraduate medical education. There has been a significant impact on students' self-directed learning, critical thinking, and self-monitoring. Using the e-portfolio is extremely advantageous to the holistic development of students-both in their personal and professional domains. It is thus imperative that activities focussing on reflective practice be introduced into formal curricular delivery in undergraduate medical education.
... Although much effort was put into the execution of this research project, there remain several limitations that should be acknowledged. Firstly, since there is probably a gap between teachers' self-report and real capacity [80,81], further research might consider triangulating data sources observations or testing to increase data objectivity. Secondly, the present study investigates teachers' capacity for Action research and the difficulties they encountered, but it does not cover insights into the forms of support for teacher Action research. ...
Action research (AR), which involve rigorous cycles of implementing and reflecting on educational changes, brings genuine values to language teaching. However, due to several factors related to school cultures and the teacher-researchers, AR does not always live up to its expectations. Generally, previous studies have steered towards instigating an awareness of AR at the expense of obtaining empirical evidence on competencies and conditions for AR. This study examines teachers’ action research competencies and challenges in doing AR. A questionnaire (70 items, α = 0.809) was administered to 105 EFL teachers in various high schools in Vietnam. Among these participants, eight partook in follow-up semi-structured interviews. Chi-square analysis confirmed that top-rated competencies aligned with the traditional roles of high school English teachers, including giving feedback, using visual aids for presentations, and self-assessment. In contrast, teachers were least sure of their research skills and techniques. Thematic analysis complemented the quantitative data, ascribing the underdeveloped research capacity to a lack of guidelines, motivation, and resources. Consequently, the practice of AR might be improved with the cultivation of a teacher research culture. Other measures to incentivize high school EFL teachers in research engagement are also discussed in this paper.
... A harmadik hipotézis feltételezi, hogy a felkészültebb hallgatók hajlamosabbak korrigálni önértékelésüket a megszerzett tapasztalatok alapján (ld. Nierenberg & Dahl 2021, Edwards et al. 2003, Eva et al. 2004. Esetünkben ez azt jelenti, hogy számíthatunk arra, hogy a vizsga után azok javítanak jelentősebb mértékben a vizsga előtt tett önértékelésükön, akik jobban teljesítenek. ...
A TANULMÁNY CÉLJA A tanulmány célja Dunning–Kruger-hatás elemzése az üzleti felsőoktatási szakképzésben, illetve annak vizsgálata, hogy az önértékelés pontossága fejleszthető-e a felkészültség emelésével. ALKALMAZOTT MÓDSZERTAN Feleltválasztásos vizsgatesztek eredményeit (N = 359) vetettük össze a hallgatók saját vizsga előtti és utáni becsléseivel, lineáris regressziós modellekben, kontroll változók szerepeltetése mellett. LEGFONTOSABB EREDMÉNYEK A gyengébb teljesítményű hallgatók hajlamosabbak saját teljesítményüket felül- és egyben pontatlanabbul értékelni. A felkészültebb hallgatók hajlamosabbak korrigálni önértékelésüket a megszerzett tapasztalatok alapján. Ezt mind a vizsgák alatt, mind két vizsga között alátámasztottuk. Mindezekben nem találtunk különbséget a nemek között. GYAKORLATI JAVASLATOK Eredményeink alapján mind a tesztírás maga, mind az alaposabb felkészülés csökkentette az önértékelési hibákat, ezért javasolható a tesztelés gyakoriságának növelése (évközi és/vagy próbatesztekkel). Köszönetnyilvánítás: A tanulmány az EFOP3.6.3-VEKOP-16-2017-00007-"Tehetségből fiatal kutató" – A kutatói életpályát támogató tevékenységek a felsőoktatásban projekt támogatásával készült.
... Although studentdirected learning has many strengths, [23,24] it does pose unique challenges to the integration of health humanities within a medical school curriculum. Many students come to medical school with a biomedical background and have limited exposure to the humanities in their undergraduate education. ...
Introduction: The humanities enrich and transform the practice of medicine. What remains to be seen, however, is how best to integrate humanities into the medical curriculum to optimize both educational and patient-related outcomes. The present study considers the structure of an innovative student-driven humanities curriculum and seeks to understand its strengths and limitations, as well as make recommendations for improvement. Results: 420 HLOs met criteria for analysis and were coded as instrumental (developing direct clinical skill), non-instrumental (non-skill based), or both. Of these, 125 (30%) were instrumental, 239 (57%) were non-instrumental, and 56 (13%) were coded as both. Most instrumental HLO centered around communication skills. Non-instrumental HLO most commonly focused on bearing witness and critiquing a particular experience within a social and/or political context. Conclusions: Findings from this study contribute to the development of a humanities curriculum in a student-directed learning program. Non-instrumental HLO lacked a theoretical framework to guide student's investigations to a deeper level of analysis. Student-directed learning offers many strengths, but can be enhanced through external direction from humanities trained faculty, particularly given that many medical students have a limited humanities background.
... While medical students themselves recognize the importance of science-based decision-making, they often do not feel confident in their ability to do so (Pruskil et al., 2009) although there is also evidence of overconfidence effects (Borracci and Arribalzaga, 2018). This, in turn, may be due to deficits in their ability to evaluate their own knowledge and recognize deficiencies therein (Kruger and Dunning, 1999;Eva et al., 2004). With regard to the learning progressions of medical students, making professional decisions and solving problems in clinical contexts require not only the acquisition of knowledge but also an increase in confidence in their medical knowledge (Khan et al., 2001). ...
... It appears that as one's competencies are degraded by the effects of distress, burnout, and secondary traumatic stress, one of the competencies being degraded is the ability to accurately self-assess (Dunning et al., 2003). Each of these challenges with self-assessment is found among trainees and practicing health professionals, including psychologists (Walfish et al., 2012), with little correlation found between these individuals' self-assessments and external ratings (Davis et al., 2006;Eva et al., 2004). These shortcomings of selfassessment underscore the importance of the feedback and insight one may glean from a trusted group of colleagues and peers, in regard to both one's ongoing functioning and in the development of one's self-care plan. ...
Clinical work as a mental health practitioner can be very rewarding and gratifying. It also may be stressful, difficult, and emotionally demanding for the clinician. Failure to sufficiently attend to one’s own functioning through appropriate ongoing self-care activities can have significant consequences for the practitioner’s personal and professional functioning to include experiencing symptoms of burnout and compassion fatigue that may result in problems with professional competence. The American Psychological Association (2017) ethics code mandates ongoing self-monitoring and self-assessment to determine when one’s competence is at risk or already degraded and the need to then take needed corrective actions. Yet research findings demonstrate how flawed self-assessment is and that many clinicians will not know when assistance is needed or what support or interventions are needed. Instead, a communitarian approach to self-care is recommended. This involves creating and actively utilizing a competence constellation of engaged colleagues who assess and support each other on an ongoing basis. Recommendations are made for creating a self-care plan that integrates both one’s independent self-care activities and a communitarian approach. The role of this approach for promoting ongoing wellness and maintaining one’s clinical competence while preventing burnout and problems with professional competence is accentuated. The use of this approach as a preventive activity as well as one for overcoming clinician biases and self-assessment flaws is explained with recommendations provided for practical steps each mental health practitioner can take now and moving forward.
... [3][4][5][6] Some authors suggest that self-assessment may even be harmful to learning if it is uninformed and uncalibrated by an external assessor. 7,8 Others have suggested that the inconsistencies between learner selfassessment and teacher judgements of competence make it inadvisable to use learner self-assessments in decision-making about competence. 9 For these reasons, it is important that learner self-assessments of progress be monitored for both accuracy (ie, consistency with assessors) and change over the course of training. ...
Background and objectives:
In competency-based medical education (CBME), should resident self-assessments be included in the array of evidence upon which summative progress decisions are made? We examined the congruence between self-assessments and preceptor assessments of residents using assessment data collected in a 2-year Canadian family medicine residency program that uses programmatic assessment as part of their approach to CBME.
Methods:
This was a retrospective observational cohort study using a learning analytics approach. The data source was archived formative workplace-based assessment forms (fieldnotes) stored in an online portfolio by family medicine residents and preceptors. Data came from three academic teaching sites over 3 academic years (2015-2016, 2016-2017, 2017-2018), and were analyzed in aggregate using nonparametric tests to evaluate differences in progress levels selected both within and between groups.
Results:
In aggregate, first-year residents' self-reported progress was consistent with that indicated by preceptors. Progress level rating on fieldnotes improved over training in both groups. Second-year residents tended to assign themselves higher ratings on self-entered assessments compared with those assigned by preceptors; however, the effect sizes associated with these findings were small.
Conclusions:
Although we found differences in the progress level selected between preceptor-entered and resident-entered fieldnotes, small effect sizes suggest these differences may have little practical significance. Reasonable consistency between resident self-assessments and preceptor assessments suggests that benefits of guided self-assessment (eg, support of self-regulated learning, program efficacy monitoring) remain appealing despite potential risks.
... Although student self-reports can be useful for characterizing a wide range of subjective qualities (Lopatto et al., 2008;Cuthbert et al., 2012;Shaffer et al., 2014), recent work has indicated self-assessments of learning do not always correspond with empirical measures, with respondents sometimes overestimating or even underestimating their improvements (Eva et al., 2004;Schiekirka et al., 2013;Ziegler and Montplaisir, 2014). In this study, the application of more rigorous methods such as an established science process skills test (Burns et al., 1985;Dirks and Cunningham, 2006;Feyzíoglu, 2012;Kramer et al., 2018) was avoided, because the endeavor would have entailed the laborious process of translation and validation. ...
Infusing undergraduate curricula with authentic research training is an important contemporary challenge. Such exposure typically occurs through mentored research (MR) or course-based undergraduate research experiences (CUREs). In Asian contexts, CURE implementation is rare, while MR is often a graduation requirement. In this study, mentor interviews and mentee focus groups were used to characterize the learning challenges associated with this requirement at a Chinese university. An intensive 6-week CURE was then implemented as an MR preparatory program to help mitigate the identified challenges. This program contained seven site-specific features not typically included in other CUREs, each designed to improve different aspects of student readiness for MR. Post-CURE surveys, focus groups, and interviews demonstrated CURE enrollment significantly improved subsequent MR outcomes. Almost 90% of all enrollees, for example, began their first MR experience in their second year, more than twice the rate of non-enrollees. Enrollees also reported greater confidence in their research skills and more frequent experiences working in multiple labs. This study reports both immediate CURE and downstream MR outcomes, using the former to help explain the latter. A comprehensive CURE implementation process is described, offering a potential model for the design of other programs with similar research enhancement goals.
... Moreover, as demonstrated in this study, self-reported competence is weakly related to performance-based competence outcomes. Consistent with other work on competency, the relationship between self-evaluated competence and performance-based competence is minimal (Davis et al., 2006;Eva et al., 2004). As such, focus on self-awareness and self-knowledge in competency development (Kaslow et al., 2018) would benefit from ensuring trainee perceptions of their competency align with benchmarked progression. ...
Objective:
Attaining competence in assessment is a necessary step in graduate training and has been defined to include multiple domains of training relevant to this attainment. While important to ensure trainees meet these standards of training, it is critical to understand how and if competence shapes a trainees' professional identity, therein promoting lifelong competency.
Methods:
The current study assessed currently enrolled graduate trainees' knowledge and perception of their capabilities related to assessment to determine if self-reported and performance-based competence would incrementally predict their intention to use assessment in their future above basic training characteristics and intended career interests.
Results:
Self-reported competence, but not performance-based competence, played an incremental role in trainees' intention to use assessments in their careers. Multiple graduate training characteristics and practice experiences were insignificant predictors after accounting for other relative predictors (i.e., intended career settings, integrated reports).
Conclusion:
Findings are discussed about the critical importance of incorporating a hybrid competency-capability assessment training framework to further emphasize the role of trainee self-efficacy in hopes of promoting lifelong competence in their continued use of assessments.
... Therefore, our participants' narratives about the aspects of training that were most beneficial for their learning should be contextualized within the literature on the limitations of self-assessment. 40 ...
Introduction:
Effective leadership improves patient care during medical and trauma resuscitations. While dedicated training programs can improve leadership in trauma resuscitation, we have a limited understanding of the optimal training methods. Our objective was to explore learners' and teachers' perceptions of effective methods of leadership training for trauma resuscitation.
Methods:
We performed a qualitative exploration of learner and teacher perceptions of leadership training methods using a modified grounded theory approach. We interviewed 28 participants, including attending physicians, residents, fellows, and nurses who regularly participated in trauma team activations. We then analyzed transcripts in an iterative manner to form codes, identify themes, and explore relationships between themes.
Results:
Based on interviewees' perceptions, we identified seven methods used to train leadership in trauma resuscitation: reflection; feedback; hands-on learning; role modeling; simulation; group reflection; and didactic. We also identified three major themes in perceived best practices in training leaders in trauma resuscitation: formal vs informal curriculum; training techniques for novice vs more senior learner; and interprofessional training. Participants felt that informal training methods were the most important part of training, and that a significant part of a training program for leaders in trauma resuscitation should use informal methods. Learners who were earlier in their training preferred more supervision and guidance, while learners who were more advanced in their training preferred a greater degree of autonomy. Finally, participants believed leadership training for trauma resuscitation should be multidisciplinary and interprofessional.
Conclusion:
We identified several important themes for training leaders in trauma resuscitation, including using a variety of different training methods, adapting the methods used based on the learner's level of training, and incorporating opportunities for multidisciplinary and interprofessional training. More research is needed to determine the optimal balance of informal and formal training, how to standardize and increase consistency in informal training, and the optimal way to incorporate multidisciplinary and interprofessional learning into a leadership in trauma resuscitation training program.
... While medical students themselves recognize the importance of science-based decision-making, they often do not feel confident in their ability to do so (Pruskil et al., 2009) although there is also evidence of overconfidence effects (Borracci and Arribalzaga, 2018). This, in turn, may be due to deficits in their ability to evaluate their own knowledge and recognize deficiencies therein (Kruger and Dunning, 1999;Eva et al., 2004). With regard to the learning progressions of medical students, making professional decisions and solving problems in clinical contexts require not only the acquisition of knowledge but also an increase in confidence in their medical knowledge (Khan et al., 2001). ...
Research Focus: The promotion of domain-specific knowledge is a central goal of higher education and, in the field of medicine, it is particularly essential to promote global health. Domain-specific knowledge on its own is not exhaustive; confidence regarding the factual truth of this knowledge content is also required. An increase in both knowledge and confidence is considered a necessary prerequisite for making professional decisions in the clinical context. Especially the knowledge of human physiology is fundamental and simultaneously critical to medical decision-making. However, numerous studies have shown difficulties in understanding and misconceptions in this area of knowledge. Therefore, we investigate (i) how preclinical medical students acquire knowledge in physiology over the course of their studies and simultaneously gain confidence in the correctness of this knowledge as well as (ii) the interrelations between these variables, and (iii) how they affect the development of domain-specific knowledge.
Method: In a pre–post study, 169 medical students’ development of physiology knowledge and their confidence related to this knowledge were assessed via paper-pencil questionnaires before and after attending physiology seminars for one semester. Data from a longitudinal sample of n = 97 students were analyzed using mean comparisons, regression analyses, and latent class analyses (LCAs). In addition, four types of item responses were formed based on confidence and correctness in the knowledge test.
Results: We found a significant and large increase in the students’ physiology knowledge, with task-related confidence being the strongest predictor (apart from learning motivation). Moreover, a significantly higher level of confidence at t2 was confirmed, with the level of prior confidence being a strong predictor (apart from knowledge at t2). Furthermore, based on the students’ development of knowledge and confidence levels between measurement points, three empirically distinct groups were distinguished: knowledge gainers, confidence gainers, and overall gainers. The students whose confidence in incorrect knowledge increased constituted one particularly striking group. Therefore, the training of both knowledge and the ability to critically reflect on one’s knowledge and skills as well as an assessment of their development in education is required, especially in professions such as medicine, where knowledge-based decisions made with confidence are of vital importance.
... Moreover, we decided to limit the number of questions asked to increase the number of participants. As self-assessments can be an unreliable proxy of actual knowledge (29), some of our survey questions may be prone to self-evaluation bias. In addition, data collection took place mostly at the beginning of the pandemic. ...
Background: The emerging field of artificial intelligence (AI) will probably affect the practice for the next generation of doctors. However, the students' views on AI have not been largely investigated.
Methods: An anonymous electronic survey on AI was designed for medical and dental students to explore: (1) sources of information about AI, (2) AI applications and concerns, (3) AI status as a topic in medicine, and (4) students' feelings and attitudes. The questionnaire was advertised on social media platforms in 2020. Security measures were employed to prevent fraudulent responses. Mann-Whitney U-test was employed for all comparisons. A sensitivity analysis was also performed by binarizing responses to express disagreement and agreement using the Chi-squared test.
Results: Three thousand one hundred thirty-three respondents from 63 countries from all continents were included. Most respondents reported having at least a moderate understanding of the technologies underpinning AI and of their current application, with higher agreement associated with being male (p < 0.0001), tech-savvy (p < 0.0001), pre-clinical student (p < 0.006), and from a developed country (p < 0.04). Students perceive AI as a partner rather than a competitor (72.2%) with a higher agreement for medical students (p = 0.002). The belief that AI will revolutionize medicine and dentistry (83.9%) with greater agreement for students from a developed country (p = 0.0004) was noted. Most students agree that the AI developments will make medicine and dentistry more exciting (69.9%), that AI shall be part of the medical training (85.6%) and they are eager to incorporate AI in their future practice (99%).
Conclusion: Currently, AI is a hot topic in medicine and dentistry. Students have a basic understanding of AI principles, a positive attitude toward AI and would like to have it incorporated into their training.
... Given the significant global changes and move to remote learning, how these results generalize and how providers' preferences have already changed is unknown. Finally, our participants' preferences should be viewed in the context of the literature on the limitations of self-assessment and judgments of learning [14]. As such, these findings should guide future studies rather than be viewed as guidance on optimizing CME delivery. ...
Study objective
We investigate the continuing medical education delivery preferences among emergency medicine providers, both physicians and advanced practice providers (APPs), within a large, national emergency medicine group.
Methods
A survey was sent via email to all emergency medicine health care providers in the practice group, including questions evaluating both delivery method and topic preference for continuing education. The study was sent to providers whom the group employed from October 2019 through January 2020.
Results
A total of 2038 providers, 1373 physicians, and 665 APPs completed the survey - a completion rate of 68.7%. In general, practitioners expressed willingness to learn across various platforms, with a strong overall preference towards online and on-demand options, including video, podcast, and written materials. Universally, a larger percentage of APPs identified a desire for more continuing education across all topics than physicians.
Conclusion
Education preferences among emergency medicine providers vary with a strong trend toward online and on-demand content. Understanding the delivery and topic preferences for providers is important for the optimal creation of continuing education content.
... Given that humans are notoriously bad at self-assessment, maintaining appropriate levels of confidence can be challenging if left entirely to the individual to resolve. [21][22][23]51,52 Rather than measuring confidence as an independent variable or construct, we can consider it as being linked to (but not as a surrogate for) competence. The interaction of confidence and competence in HPE has long been considered, [53][54][55][56][57][58] and individuals are viewed to be problematic when confidence and competence are decoupled. ...
Context
There have been significant advances in competency-based medical education (CBME) within health professions education. While most of the efforts have focused on competency, less attention has been paid to the role of confidence as a factor in preparing for practice. This paper seeks to address this deficit by exploring the role of confidence and the calibration of confidence with regard to competence.
Methods
This paper presents a conceptual review of confidence and the calibration of confidence in different medical education contexts. Building from an initial literature review, the authors engaged in iterative discussions exploring divergent and convergent perspectives, which were then supplemented with targeted literature reviews. Finally, a stakeholder consultation was conducted to situate and validate the provisional findings.
Results
A series of axioms were developed to guide perceptions and responses to different states of confidence in health professionals: 1) confidence can shape how we act and is optimized when it closely corresponds to reality; 2) self-confidence is task-specific, but also inextricably influenced by the individual self-conceptualization, the surrounding system, and society; 3) confidence is shaped by many external factors and the context of the situation; 4) confidence must be considered in conjunction with competence; and 5) the confidence-competence ratio (CCR) changes over time. It is important to track learners’ CCRs and work with them to maintain balance.
Conclusion
Confidence is expressed in different ways and is shaped by a variety of modifiers. While CBME primarily focuses on competency, proportional confidence is an integral component in ensuring safe and professional practice. As such, it is important to consider both confidence and competence, as well as their relationship in CBME. The CCR can serve as a key construct in developing mindful and capable health professionals. Future research should evaluate strategies for assessing CCR, identify best practices for teaching confidence and guiding self-calibration of CCR, and explore the role of CCR in continuing professional development for individuals and teams.
Purpose
To examine the perceived self‐efficacy of dentists who recently completed Graduate Dental Education (GDE) programs and identify how closely it aligns with their supervisors’ assessments of them. Self‐efficacy has been associated with academic pursuits, motivation, and engagement, which may affect how dental providers practice, seek continuing education, and pursue future opportunities.
Methods
Recent graduates of military GDE programs rated their self‐efficacy on specific tasks within each of the seven domains of dental competencies. Their supervisors completed a similar survey, rating the graduate's performance in the same tasks. Graduates’ mean ratings were calculated for each domain, spearman correlations were calculated for all graduate‐supervisor task ratings, and the magnitude of differences between graduate and supervisor domain means were examined.
Results
Graduates’ perceived self‐efficacy ranged from 3.57 to 4.41 out of 5.0. Correlations for each task were universally weak (ρ = ‐0.04–0.27). Correlations for domain means were also weak (ρ = 0.06–0.14). Overall, graduates rated themselves lower than their supervisors, with mean differences ranging from ‐0.17 ( p = 0.003, Cohen's d = 0.20) for Professionalism to ‐0.95 ( p < 0.001, Cohen's d = 0.90) for Health Promotion .
Conclusions
Overall, graduates’ perceived self‐efficacy was moderate to high for 26 tasks across seven domains. However, in aggregate, graduates underestimated their abilities compared to performance measures from their current supervisors, although effect sizes were small. The accuracy of graduates’ self‐efficacy varied by program length and the clinical specialty of their supervisors. High‐performing graduates always underestimated themselves while low‐performing graduates often overestimated themselves.
Self-monitoring is essential for effectively regulating learning, but difficult in visual diagnostic tasks such as radiograph interpretation. Eye-tracking technology can visualize viewing behavior in gaze displays, thereby providing information about visual search and decision-making. We hypothesized that individually adaptive gaze-display feedback improves posttest performance and self-monitoring of medical students who learn to detect nodules in radiographs. We investigated the effects of: (1) Search displays, showing which part of the image was searched by the participant; and (2) Decision displays, showing which parts of the image received prolonged attention in 78 medical students. After a pretest and instruction, participants practiced identifying nodules in 16 cases under search-display, decision-display, or no feedback conditions (n = 26 per condition). A 10-case posttest, without feedback, was administered to assess learning outcomes. After each case, participants provided self-monitoring and confidence judgments. Afterward, participants reported on self-efficacy, perceived competence, feedback use, and perceived usefulness of the feedback. Bayesian analyses showed no benefits of gaze displays for post-test performance, monitoring accuracy (absolute difference between participants’ estimated and their actual test performance), completeness of viewing behavior, self-efficacy, and perceived competence. Participants receiving search-displays reported greater feedback utilization than participants receiving decision-displays, and also found the feedback more useful when the gaze data displayed was precise and accurate. As the completeness of search was not related to posttest performance, search displays might not have been sufficiently informative to improve self-monitoring. Information from decision displays was rarely used to inform self-monitoring. Further research should address if and when gaze displays can support learning.
In the present study, we recorded the eye movements of 20 criterion-based selected trainee teachers in economics while they responded to 25 single choice (SC) items in an economic content knowledge (CK) test and rated their confidence for each response in a digital assessment. By using a multilevel modeling approach with crossed random effects, we confirmed prior findings from eye-tracking research on SC tests, which showed longer dwell time on the correct response options (attractor) and shorter dwell time on the distractors are positively linked to correct options. Furthermore, we identified an additional effect on dwell time on the attractor in a moderator model with participants who highly rated their confidence for correct response options. Thus, we identified a specific role of students’ confidence in their CK on the gaze bias effect. We interpret these results in terms of students’ actual understanding of test contents from assessments of their professional knowledge and draw implications for further research and teacher education.
Introduction
Self‐monitoring of clinical‐decision‐making is essential for health care professional practice. Using certainty in responses to assessment items could allow self‐monitoring of clinical‐decision‐making by medical students to be tracked over time. This research introduces how aspects of insightfulness, safety and efficiency could be based on certainty in, and correctness of, multiple‐choice question (MCQ) responses. We also show how these measures change over time.
Methods
With each answer on twice yearly MCQ progress tests, medical students provided their certainty of correctness. An insightful student would be more likely to be correct for those answers given with increasing certainty. A safe student would be expected to have a high probability of being correct for answers given with a high certainty. An efficient student would be expected to have a sufficiently low probability of being correct when they have no certainty. The system was developed using first principles and data from one cohort of students. A dataset from a second cohort was then used as an independent validation sample.
Results
The patterns of aspects of self‐monitoring were similar for both cohorts. Almost all the students met the criteria for insightfulness on all tests. Most students had an undetermined outcome for the safety aspect. When a definitive result for safety was obtained, absence of safety was most prevalent in the middle of the course, while the presence of safety increased later. Most of the students met the criteria for efficiency, with the highest prevalence mid‐course, but efficiency was more likely to be absent later.
Discussion
Throughout the course, students showed reassuring levels of insightfulness. The results suggest that students may balance safety with efficiency. This may be explained by students learning the positive implications of decisions before the negative implications, making them initially more efficient, but later being more cautious and safer.
Introduction
Selecting medical students for residency is a competitive process, with a narrow range of scores separating middle-ranked applicants. Self-assessment is a fundamental skill for any competent physician with a demonstrated correlation to diagnostic ability, examination scores, and technical skills, but has yet to be investigated in residency selection. The objective of this study was to investigate the relationship between self-assessment and interview performance as a potential adjunct to discriminate between applicants.
Methods
At the University of Ottawa in 2020, 55 applicants completed a 9-station interview circuit assessing different characteristics or skills important for a career in general surgery, followed by a self-assessment questionnaire evaluating their perceived performance at each station. Pearson's correlation was used to determine the relationship between self-assessment scores (SASs) and interviewer scores (ISs).
Results
There was a negative correlation between SASs and ISs for all interview stations. High performers underestimated their interview performance, and low performers overestimated their performance. Seven of the nine stations reached statistical significance (r = 0.60-0.73, P < 0.001). There was significant variability in the SAS of middle-ranked applicants, with a range three times greater than the range of ISs and demonstrating distinct self-assessment skills in candidates with very similar scores.
Conclusions
Although we strive to select applicants who will succeed in residency to become competent physicians, self-assessment skills may be a useful adjunct during the interview process to assist in discriminating between applicants with similar scores.
Introduction
The objective of this study was to compare faculty member evaluations to student self-evaluations in a clinical endodontic course in the dental school at King Saud University and to evaluate the reliability of the students’ self-assessment scores after using a rubric with well-defined criteria.
Materials and methods
Evaluated and self-evaluated endodontic cases that were clinically treated by fourth-year undergraduate dental students at the College of Dentistry, Girls University Campus (GUC), at King Saud University over two years (2017-2018) were included. Cases included anterior teeth, premolars, and molars. The evaluation form was divided into six sections with well-defined criteria to cover all aspects of nonsurgical root canal treatment with a maximum grade of 10 points can be scored for each student per case. The students evaluated themselves for each section, and then were evaluated by two faculty members. Student and faculty assessment agreement and the reliability of the students’ self-assessment scores were measured. A P-value ≤ 0.05 was considered significant.
Results
A total of 363 cases were included: 26.7% anterior teeth, 38.84% premolars, and 34.43% molars. The students evaluated themselves with higher grades compared to the evaluators’ grading in all steps and in the overall grading in all teeth types. The students’ self-assessment scores showed good and moderate reliability in all steps and in the overall grading.
Conclusion
The students tend to overrate their performance and their assessments have moderate to good reliability, which reflects the reliability of the rubric used as an accurate measurement tool that helps the evaluator and the student objectively assess their performance.
Background
Many medical schools offer a culminating internship readiness experience. Curricula focus on particular knowledge and skills critical to internship, such as answering urgent nursing pages. Studies have shown student performance improvement with mock paging education programs, but the role of feedback versus self-regulated practice has not been studied.Design and Methods
The interprofessional mock paging program included 156 medical students enrolled in a 4th-year internship readiness course and 44 master’s level direct entry nursing students. Medical students were randomized to receive verbal feedback immediately after each of the three phone calls (intervention group) or delayed written feedback (control group) after the third phone call only. Specialty-specific case scenarios were developed and a single checklist for all scenarios was developed using the communication tool ISBAR. Medical students and nursing students had separate training sessions before the pages commenced. The nursing students administered the phone calls and evaluated the medical students by ISBAR checklist. An interrater reliability measure was obtained with physician observation of a selection of phone calls.ResultsAfter adjusting for the case effects (different case scenarios for different specialties), students showed no statistically significant differences on checklist scores for case 1 (first case, F = 1.491, df = 1, p = .224), but did show statistically significant differences on checklist scores for case 3 (final case, F = 12.238, df = 1, p = .001). Strong interrater reliability was found between the faculty physician and observed nursing students (ICC = .89).Conclusions
Immediate feedback significantly improves student checklist scores with a mock paging program. This finding suggests that coaching with feedback may have advantages above self-regulated learning.
Self-assessment, or students’ evaluation of their own learning or performance in academic tasks, is a self-regulatory process that is intertwined with learners’ cognitive, behavioral, and motivational processes. Among the motivational beliefs that have been studied in relation to self-assessment is self-efficacy, which refers to beliefs about one's capability to learn or perform a specified task successfully. The current study describes the development and validation of the Self-Efficacy for Self-Assessment in Argumentative Writing (SEESA-AW) scale to measure college students’ self-efficacy beliefs for self-assessment in the domain of argumentative writing with two samples of undergraduate college students (N=335 and N=662). Exploratory and confirmatory factor analyses revealed three factors (grammar & mechanics, cohesiveness/flow, the presentation of arguments) that explained 65.4% of the variance. Evidence is also provided for convergent validity of the scale with regard to other writing self-efficacy scales. Educational implications are discussed.
Background:
Electromagnetic navigation bronchoscopy (ENB) is a relatively new and technically demanding procedure for the guidance of bronchoscopic biopsy to help locate small lung lesions. The results in experienced hands are well described. However, we do not know the results in unexperienced hands-in other words, we have no knowledge about how fast you can learn the procedure.
Aim:
The aim of this study was to draw learning curves for beginners in ENB using Cumulated Sum (CUSUM) analysis, a method for quantitative evaluation of the learning curves for clinical procedures.
Methods:
Four operators from 3 centers without prior ENB experience were enrolled. The outcome of consecutive ENB procedures was assessed and classified as either diagnostic or nondiagnostic based on sample adequacy. Learning curves were drawn based on diagnostic yield.
Results:
A total of 215 procedures were assessed. For 2 of the operators (operators 1 and 4), at least 25 to 30 procedures were necessary to obtain competency whereas operators 2 and 3 showed more horizontal learning curves indicating an overweight of diagnostic procedures from the beginning.
Conclusion:
Operators achieve competences in ENB at different paces. This must be taken in account when beginners start to learn the procedure. There is a huge need for a structured educational program and a validated test to determine competences.
The purpose of this paper is to study the effectiveness of oral presentation as an assessment tool in a Finance subject. Assessment data collected from a postgraduate Finance subject in an Australian university over a period of five years from 2005 to 2009 was analysed statistically to determine the relation between students’ performance in oral presentation and other forms of assessments. The sample consists of assessment records of 412 students and 98 group presentations. From the study of correlations between oral presentations and other assessments, it is concluded that students perform better in written assessments compared to oral assessment. The study of effect of gender on students’ performance leads to the conclusion that female students perform better than male students in all forms of assessments except oral presentations where male students perform better although difference between males and females in oral presentation is not very large. The study of students’ performance based on their nationality leads to the conclusion that domestic students perform better than international students in all forms of assessments. Based on the study of student’ performance in oral presentation, it is found that students did well in the development of content of presentations, quality of their analysis, group coordination and organisation of presentation. There is however a general tendency to treat group work as a sum of parts instead of treating the group work as a single task. This study is limited by the fact that effectiveness of oral presentation is studied in only one Finance subject. This study makes an original contribution to the literature as the effectiveness of oral assessment in Finance subject is being studied for the first time. The conclusions arrived in this paper have many implications for policies and practice of learning and teaching in Finance.
To compare how well graduates of a self-directed, problem-based undergraduate curriculum (at McMaster University [MU], Hamilton, Ont.) and those of a traditional curriculum (at the University of Toronto [UT]) who go on to primary care careers keep up to date with current clinical practice guidelines.
Analytic survey. Management of hypertension was chosen as an appropriate topic to assess guideline adherence. An updated version of a previously validated questionnaire was mailed to the participants for self-completion.
Private primary care practices in southern Ontario.
A random sample of 48 MU graduates and 48 UT graduates, stratified for year of graduation (1974 to 1985) and sex, who were in family or general practice in Ontario; 87% of the eligible subjects in each group responded.
Overall and component-specific scores; analysis was blind to study group.
The overall mean scores were 56 (68%) of a possible 82 for the MU graduates and 51 (62%) for the UT graduates (difference between the means 5, 95% confidence interval 1.9 to 8.2; p < 0.01). Multivariate linear regression analysis of seven factors that might affect questionnaire scores revealed that only the medical school was statistically significant (p < 0.01). The MU graduates had significantly higher mean scores than the UT graduates for two components of the questionnaire: knowledge of recommended blood pressures for treatment (p < 0.05) and successful approaches to enhance compliance (p < 0.005). The trends were similar for the other components but but were not significant.
The graduates of a problem-based, self-directed undergraduate curriculum are more up to date in knowledge of the management of hypertension than graduates of a traditional curriculum.
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
The competent physician pursues lifelong learning through the recognition of deficiencies and the formulation of appropriate learning goals. Despite the accepted theoretical value of self-assessment, studies have consistently shown that the accuracy of self-assessment is poor. This paper examines the methodological issues that plague the measurement of self-assessment ability and presents several strategies that address these methodological problems within the current paradigm. In addition, the article proposes an alternative conceptualization of self-assessment and describes its associated methods. The conclusions of prior research in this domain must be re-examined in light of the common pitfalls encountered in the design of the studies and the analyses of the data. Future efforts to elucidate self-assessment phenomena need to consider the implications of this review.
People feel they understand complex phenomena with far greater precision, coherence, and depth than they really do; they are subject to an illusion-an illusion of explanatory depth. The illusion is far stronger for explanatory knowledge than many other kinds of knowledge, such as that for facts, procedures or narratives. The illusion for explanatory knowledge is most robust where the environment supports real-time explanations with visible mechanisms. We demonstrate the illusion of depth with explanatory knowledge in Studies 1-6. Then we show differences in overconfidence about knowledge across different knowledge domains in Studies 7-10. Finally, we explore the mechanisms behind the initial confidence and behind overconfidence in Studies 11 and 12. Implications for the roles of intuitive theories in models of concepts and cognition are discussed.
Although expected of all health professionals, self-assessment skills are seldom addressed directly in training. A previous review by the author identified curricular criteria associated with improved accuracy and validity in self-assessments of knowledge and performance in curriculum studies published between January 1970 and February 1990. The present review analyzed 11 studies that meet those criteria. Eight studies were of implementations of self-assessment components within training programs in the health professions, and three involved other training environments. Most described initial disorientation or opposition on the part of learners, attributed to unfamiliar roles and to learners' distrust. The curricula that successfully negotiated the transition to self-assessment norms reported noncognitive benefits such as improvements in morale, motivation, and communications among learners and faculty. Reported cognitive benefits included improvements in knowledge, performance, and self-analysis of performance. The constellation of effects suggests that effective self-assessment programs may promote more mature, collegial, and productive learning environments, particularly suited to the training of health professionals. Most curricula fostering effective self-assessment did not require extraordinary resources, and none jeopardized traditional standards. No evidence was found to support or challenge the expectation that self-assessment training would transfer to later work settings.
Valid self-assessment is fundamental to continuing professional competence but is seldom explicitly taught in health professions training. This review analyzed 18 scholarly articles published between January 1970 and February 1990 (14 articles regarding health professions trainees, and four concerning college students or graduate trainees) in which it was possible to compare performance as self-assessed by trainees with performance as assessed by experts or objective tests. The validity of self-assessed performance was found to be low to moderate and did not improve with time in conventional health professions training programs. Self-assessed performance seemed closely related to generalized self-attributions and was minimally influenced by external feedback in the form of test scores, grades, or faculty assessments. In five programs emphasizing explicit self-assessment goals and training strategies, moderate-to-high validity outcomes or improvements over time were demonstrated. Much of what passes for self-assessment in training seems the exercise of an underdeveloped skill, but effective training to improve validity and accuracy is available and feasible.
Medical students in a tutorial program rated themselves in Year 1 and again in Year 2 on Knowledge, Reasoning, Communication and Interpersonal Skills. Student and tutor ratings were similar in Year 1 but different in Year 2. Both self and tutor ratings increased from Year 1 to Year 2 but self-ratings were independent of achievement while tutor ratings were related to achievement. Possible reasons for these results were discussed with reference to the learning goals of the tutorials. © 1989 Informa UK Ltd All rights reserved: reproduction in whole or part not permitted.
The purpose of this paper was to determine if the decline in performance with time since completion of training on the 1980 ABIM Recertification Examination can be explained by a difference in performance on items testing different types of knowledge. Results showed that candidates further out of training performed less well on items testing new or changing knowledge, while performance on items testing stable knowledge was relatively constant across age groups.
Although self-evaluation is crucial in the practice of medicine, few educators have formally introduced self-assessment into the undergraduate medical curriculum. However, students in the baccalaureate-M.D. degree program at the University of Missouri, Kansas City, must complete a self-evaluation at the close of every medical school course and rotation during the last four years of a six-year curriculum. In this paper, the authors examine the self-ratings of 211 of these students as they progressed through the program in order to discover trends in and correlates of the self-assessments. Although the students' self-evaluations and faculty members' ratings of these students' performances rose year by year, the relationship between the students' and the faculty's ratings decreased through time. Yet, results suggest that self-evaluation has educational merit as a measure of noncognitive abilities associated with clinical performance and as a stimulus to further learning and professional development.
To determine whether continuing medical education affects the quality of clinical care, we randomly allocated 16 Ontario family physicians to receive or not receive continuing-education packages covering clinical problems commonly confronted in general practice. Over 4500 episodes of care, provided before and after study physicians received continuing education, were compared with preset clinical criteria and classified according to quality. Although objective tests confirmed that the study physicians learned from the packages, there was little effect on the overall quality of care. When the topics were of relatively great interest to the physicians, the control group (who did not receive the packages) showed as much improvement as did the study group. When the topics were not preferred, however, the documented quality of care provided by study physicians rose (P less than 0.05) and differed from that provided by control physicians (P = 0.01). Finally, there was no spillover effect on clinical problems not directly covered by the program. In view of the trend toward mandatory continuing education and the resources expended, it is time to reconsider whether it works.
As physicians, whether serving individual patients or populations, we always have sought to base our decisions and actions on the best possible evidence. The ascendancy of the randomized trial heralded a fundamental shift in the way that we establish the clinical bases for diagnosis, prognosis, and therapeutics. The ability to track down, critically appraise (for its validity and usefulness), and incorporate this rapidly growing body of evidence into one's clinical practice has been named 'evidence-based medicine' (EBM).
The goal of the study was to examine critically the use of taxonomic level as a criterion for determining the utility of a test item. If the distinction is useful, several hypotheses follow: There should be reasonable agreement among raters about taxonomic level; the taxonomic level should be independent of the level of the rater; in contrasting students from PEL and conventional curricula, since PEL students consistently report that they spend more time on conceptualizing and less on memorizing, there should be differential performance on factual recall and higher order items; higher order items should have greater predictive validity (addressed in other studies). Test items from the 1993 and 1994 Medical Council of Canada multiple choice examinations were reviewed by two recent graduates and two faculty. Although there was moderate agreement among raters, faculty were significantly more likely to rate items as Factual Recall. Performance of McMaster students and other Canadian graduates were then compared at an item level. No systematic differences between the two cohorts related to taxonomic level were identified. In conclusion, while there is moderate agreement about the taxonomic level of a test item, this appears to be related to educational level, and has little heuristic value as an indicator of performance.
The progress test (or Quarterly Profile Examination), invented concurrently by the University of Missouri-Kansas City School of Medicine and the University of Limburg, is used to provide useful summative and formative judgments about students' knowledge without distorting learning. All students in all classes sit the same examination at regular intervals through the year, and their individual progress is noted. This paper reports on four years' experience with a progress test, the Personal Progress Index (PPI), at the McMaster University Faculty of Health Sciences.
The PPI, a 180-item multiple-choice test with items drawn from all disciplines of medicine, is administered to medical students in all three classes three times per year. Individual feedback is provided, and accumulated student performance is determined for summative purposes. This paper examines extensive evidence of reliability, validity, and effect on student learning, using samples from the entering classes of 1992-1995 (a total of 400 students).
Reliabilities across test administrations (within classes) ranged from .46 to .63. The PPI demonstrated strong construct validity, with highly significant statistical tests of differences between classes and changes within classes on successive administrations. The predictive validity of the PPI, i.e., whether it could predict performance on the licensing examination of the Medical Council of Canada, increased monotonically from a correlation of .12 for the first test administration one month into medical school to a high of about .60 for the cumulative score across all administrations three months prior to the examination.
The PPI seems to be performing as intended, with students showing growth in performance across the three years of the MD program. Additional benefits are that many more students now self-refer for remediation (based on low PPI scores) and that the consistent relative performances of individual students across test administrations allow for the identification of students who have severe and persistent problems.
Although studies have examined medical students' ability to self-assess their performance, there are few longitudinal studies that document the stability of self-assessment accuracy over time. This study compares actual and estimated examination performance for three classes during their first 3 years of medical school.
Students assessed their performance on classroom examinations and objective structured clinical examination (OSCE) stations. Each self-assessment was then contrasted with their actual performance using idiographic (within-subject) methods to define three measures of self-assessment accuracy: bias (arithmetic differences of actual and estimated scores), deviation (absolute differences of actual and estimated scores), and covariation (correlation of actual and estimated scores). These measures were computed for four intervals over the course of 3 years. Multivariate analyses of variance and correlational analyses were used to evaluate the stability of these measures.
Self-assessment accuracy measures were relatively stable over the first 2 years of medical school with a decease occurring in the third year. However, the correlational analyses indicated that the stability of self-assessment accuracy was comparable to the stability of actual performance over this same period.
The apparent decline in accuracy in the third year may reflect the transition from familiar classroom-based examinations to the substantially different clinical examination tasks of the third year OSCE. However, the stability of self-assessment accuracy compares favorably with the stability of actual performance over this period. These results suggest that self-assessment accuracy is a relatively stable individual characteristic that may be influenced by task familiarity.
The ends, and beginnings, or science in the head: coming to grasp the causal structure of the world
- F Keil