The Validity and Reliability of the Affective Competency Score to Evaluate Death Disclosure Using Standardized Patients

Department of Emergency Medicine, Emory University School of Medicine, Atlanta, GA 30303, USA.
Journal of Palliative Medicine (Impact Factor: 1.91). 05/2006; 9(2):361-70. DOI: 10.1089/jpm.2006.9.361
Source: PubMed


To explore the validity and reliability of the affective competency score (ACS), compared to a global rating measure to predict overall competency to perform a death disclosure in a standardized patient exercise and to investigate useful thresholds of the ACS.
Thirty-seven fourth-year students underwent standardized patient training in death disclosure during a fourth-year emergency medicine clerkship. Students were evaluated using a checklist, an ACS, and a global rating assessment. ACS interrater reliability, interitem reliability, item-total reliability, and split-half reliability were calculated. Area under the curve (AUC) measurements were used to establish criterion validity.
For the ACS, item-total correlations ranged from 0.76 to 0.85, 0.76 to 0.93, and 0.42 to 0.87; the split-half reliability was 0.82 (p = 0.0001), 0.86 (p = 0.0001) and 0.55 (p = 0.0007) for the standardized patient (SP), the faculty and the medical students, respectively. Interitem correlations were adequate. A moderate interrater correlation of the ACS was observed between the faculty observer and the SP (r = 0.47; p = 0.04); however, the medical students' self evaluation did not correlate significantly with either the SP (r = -0.04; p = 0.79), or the faculty observer (r = 0.00; p = 0.99). The AUC for was 0.98 (95% confidence interval [CI] 0.94 to 1.00), 0.87 (95% CI 0.73 to 0.99), and 0.74 (95% CI 0.53 to 0.95) for the faculty, SP, and medical student, respectively.
The ACS may be a valid, reliable, and useful measure to assess communication skills by faculty or SPs in this setting. At an ACS score of 16, 19, and 21 points for faculty, SPs, and medical students, respectively, there is 100% specificity for the detection of competency assessed on a global rating. However, the ACS appears to have limited reliability and validity when used by medical students.

6 Reads
  • [Show abstract] [Hide abstract]
    ABSTRACT: Delivering bad news to patients is an essential communication skill for physicians. Educators commonly use standardized patient (SP) encounters to train physicians in the delivery of bad news. It is expensive to use actors, for logistical reasons such as travel and scheduling, and there are limits to the characters and conditions an actor can portray in teaching encounters. Thus we studied the feasibility of creating SP avatars in a virtual world for the task of training medical trainees to deliver bad news. The SP avatars are easily customized for different scenarios and amenable to distance learning. We recruited 10 medical trainees to interact with a standardized female avatar in a three-dimensional simulated clinic, where the trainee was to inform the avatar of her newly diagnosed breast cancer. The trainee evaluated his or her self-efficacy in delivering bad news via an affective competency score (ACS) before and after the encounter. Two palliative care specialists evaluated each trainee's performance using the Bad News Assessment Schedule and the performance ACS. The self-efficacy ACS scores of the trainees improved overall: before, 20 ± 4, versus after, 24 ± 3, p = 0.001 (maximum score = 30). All participants considered the experience positive and commended the novel approach, although noting that the avatars were not able to portray body language cues. Participants viewed the avatar-mediated training as an excellent approach for learning how to deliver bad news but believed it could not substitute for real patient interactions. However, participant self-efficacy improved, which suggests that avatar-mediated training in a virtual world is a viable educational approach for skill training in delivering bad news.
    No preview · Article · Dec 2010 · Journal of palliative medicine
  • [Show abstract] [Hide abstract]
    ABSTRACT: To investigate changes of different domains of breaking bad news (bbn) competences after a teaching module for medical students, and to collage the results generated by different approaches of evaluation. Rating of medical student-SP interactions by means of a global rating scale and a detailed checklist used by SPs and independent raters. Students improved their breaking bad news competency. However, the changes vary between the different domains of bbn competency. In addition, results generated by different evaluation instruments differ. This study serves as a stimulus for further research on the training of specific elements of bbn and different approaches of evaluating bbn competency. In light of the different facets of bbn competency, it is important to set priorities regarding the teaching aims and to provide a consistent approach.
    No preview · Article · May 2011 · Patient Education and Counseling
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Self-assessment is an important component of medical education. Meta-analyses were conducted to better understand accuracy of self-assessment and direction of inaccuracy. Three meta-analyses were conducted on results from 35 published articles on medical student self-assessment, one for each of the theoretically distinct ways of measuring accuracy of self-reported ability (correlational, paired comparison, and independent means comparison). Characteristics that potentially influence self-assessment accuracy, including gender, year in medical school, and type of self-assessment, were examined. Students are moderately able to self-assess performance and are more accurate later in medical school. Students as a whole do not significantly over- or underestimate, but are more likely to overestimate on communication-based, standardized patient encounters than objective, knowledge-based performance measures. Female students underestimate their performance more than male students, but gender analyses are often unreported. A deeper understanding of the causes and consequences of over- and underestimation is impossible without measurement and reporting of the direction of inaccuracy. To improve our understanding of self-assessment and increase its effectiveness as a teaching tool, research should report self-assessment as both a correlation and a paired comparison, and conduct analyses of important moderators that can influence self-assessment accuracy.
    Full-text · Article · Jul 2011 · Patient Education and Counseling
Show more