Understanding self-assessment as an informed process: residents’ use of external information for self-assessment of performance in simulated resuscitations

Department of Pediatrics, University of California Davis, 2516 Stockton Blvd, Sacramento, CA, 95817, USA, .
Advances in Health Sciences Education (Impact Factor: 2.71). 03/2012; 18(2). DOI: 10.1007/s10459-012-9363-2
Source: PubMed

ABSTRACT Self-directed learning requires self-assessment of learning needs and performance, a complex process that requires collecting and interpreting data from various sources. Learners' approaches to self-assessment likely vary depending on the learner and the context. The aim of this study was to gain insight into how learners process external information and apply their interpretation of this information to their self-assessment and learning during a structured educational activity. The study combined quantitative performance data with qualitative interview data. Pediatric residents led video-recorded simulated resuscitations and rated their crisis resource management skills on a validated 6-item instrument. Three independent observers rated the videos using the same instrument. During semi-structured interviews, each resident reviewed the video, rerated performance, discussed the self-assessment process, and interpreted feedback and observer scores. Transcripts were analyzed for themes. Sixteen residents participated. Residents' self-assessed scores ranged widely but usually fell within two points of the observers. They almost universally lowered their scores when self-assessing after the video review. Five major themes emerged from qualitative analysis of their interviews: (1) residents found self-assessment important and useful in certain contexts and conditions; (2) residents varied in their self-directed learning behaviors after the simulated resuscitation; (3) quantitative observer assessment had limited usefulness; (4) video review was difficult but useful; and (5) residents focused on their weaknesses and felt a need for constructive feedback to enhance learning. The residents in our study almost uniformly embraced the importance of self-assessment for all medical professionals. Even though video review had a negative impact on their self-assessment scores and was perceived as painful, residents saw this as the most useful aspect of the study exercises residents. They were less accepting of the quantitative assessment by observers. Residents explained their tendency to focus on weaknesses as a way to create an incentive for learning, demonstrating that self-assessment is closely linked to self-directed learning. How learners can use video review and external assessment most effectively to guide their self-directed learning deserves further study.

  • [Show abstract] [Hide abstract]
    ABSTRACT: Responsiveness to feedback is a complex phenomenon that requires and receives attention. However, knowledge on the responsiveness of faculty members to residents' feedback on their teaching performance is lacking. Excellent teaching performance is essential to ensure patient safety and residents' learning in residency training. This study aims to increase our understanding of how faculty staff react to and act upon residents' feedback on their teaching performance. This multi-specialty, multi-institution interview study was conducted to gain insight into: (i) how teaching faculty proceed after they have received residents' feedback on their teaching performance, and (ii) the factors that influence their progression. Between August and December 2011, 24 faculty members who had received formative feedback on their teaching performance through valid and reliable feedback systems participated in this study. They reflected upon their (re)action(s) during individual semi-structured interviews. The interview protocol and analysis were guided by a comprehensive transtheoretical framework describing and explaining stages and processes of behavioural change. Faculty staff involved in residency training used residents' feedback to different extents to adapt or improve their teaching performance. Important tipping points in the processes of change necessary for faculty staff to put feedback into practice were: experiencing negative emotions in themselves or recognising those in residents as a result of failure to act upon feedback; realising that something should be done with or without support from others, and making a strong commitment to change. In addition, having the confidence to act upon feedback and recognising the benefits of change were found to stimulate faculty members to change their teaching behaviour. The responsiveness of faculty members to residents' feedback on their teaching performance varies. The adapted transtheoretical framework explains how and why faculty members do or do not proceed to action after receiving residents' feedback. Given this, organising residents' feedback for faculty staff in a systematic way is a first step and is necessary to effect potential improvements in teaching performance.
    Medical Education 11/2013; 47(11):1089-1098. DOI:10.1111/medu.12257 · 3.62 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Effective patient communication is correlated with better health outcomes and patient satisfaction, but is challenging to train, particularly with difficult clinical scenarios such as loss of sight. In this pilot study, we evaluated the use of simulated patient encounters with actors to train optometric students. Students were recorded during encounters with actors and assigned to an enrichment group performing five interactions with instructor feedback (n = 6) or a no-enrichment group performing two interactions without feedback (n = 4). Student performance on first and last encounters was scored with (1) subjective rating of performance change using a visual analog scale (anchors: much worse/much better), (2) yes/no response: Would you recommend this doctor to a friend/relative?, and (3) average score on questions from the American Board of Internal Medicine (ABIM) assessment of doctor communication skills. Three clinical instructors, masked to student group assignments and the order of patient encounters they viewed, provided scores in addition to self-evaluation by students and patient-actors. Using the visual analog scale, students who received enrichment were rated more improved than the no-enrichment group by masked examiners (+18 vs. -11% p = 0.04) and self-evaluation (+79 vs. +27% p = 0.009), but not by actors (+31 vs. +43%). The proportion of students recommended significantly increased following enrichment for masked examiners (61% vs. 94%; p < 0.001), but not actors (100 vs. 83%). Average ABIM assessment scores were not significantly different by any rating group: masked instructors, actors, or self-ratings. The findings of this study suggest five simulated patient encounters with feedback result in measurable improvement in student-patient communication skills as rated by masked examiners.
    Optometry and vision science: official publication of the American Academy of Optometry 11/2013; DOI:10.1097/OPX.0000000000000112 · 2.04 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Data regarding knowledge acquisition during residency training are sparse. Predictors of theoretical learning quality, academic career achievements and evidence-based medical practice during residency are unknown. We performed a cross-sectional study on residents and attending physicians across several residency programs in 2 French faculties of medicine. We comprehensively evaluated the information-seeking behavior (I-SB) during residency using a standardized questionnaire and looked for independent predictors of theoretical learning quality, academic career achievements, and evidence-based medical practice among I-SB components using multivariate logistic regression analysis. Between February 2013 and May 2013, 338 fellows and attending physicians were included in the study. Textbooks and international medical journals were reported to be used on a regular basis by 24% and 57% of the respondents, respectively. Among the respondents, 47% refer systematically (4.4%) or frequently (42.6%) to published guidelines from scientific societies upon their publication. The median self-reported theoretical learning quality score was 5/10 (interquartile range, 3-6; range, 1-10). A high theoretical learning quality score (upper quartile) was independently and strongly associated with the following I-SB components: systematic reading of clinical guidelines upon their publication (odds ratio [OR], 5.55; 95% confidence interval [CI], 1.77-17.44); having access to a library that offers the leading textbooks of the specialty in the medical department (OR, 2.45, 95% CI, 1.33-4.52); knowledge of the specialty leading textbooks (OR, 2.12; 95% CI, 1.09-4.10); and PubMed search skill score ≥5/10 (OR, 1.94; 95% CI, 1.01-3.73). Research Master (M2) and/or PhD thesis enrolment were independently and strongly associated with the following predictors: PubMed search skill score ≥5/10 (OR, 4.10; 95% CI, 1.46-11.53); knowledge of the leading medical journals of the specialty (OR, 3.33; 95% CI, 1.32-8.38); attending national and international academic conferences and meetings (OR, 2.43; 95% CI, 1.09-5.43); and using academic theoretical learning supports several times a week (OR, 2.23; 95% CI, 1.11- 4.49). This study showed weaknesses in the theoretical learning framework during residency. I-SB was independently associated with quality of academic theoretical learning, academic career achievements, and the use of evidence-based medicine in everyday clinical practice. Study registration: CNIL No.1797639.