Is Confidence of Mammographic Assessment a Good Predictor of Accuracy?

Department of Family Medicine, University of Vermont, 1 S Prospect St, Burlington, VT 05401-3444, USA.
American Journal of Roentgenology (Impact Factor: 2.73). 07/2012; 199(1):W134-41. DOI: 10.2214/AJR.11.7701
Source: PubMed


Interpretive accuracy varies among radiologists, especially in mammography. This study examines the relationship between radiologists' confidence in their assessments and their accuracy in interpreting mammograms.
In this study, 119 community radiologists interpreted 109 expert-defined screening mammography examinations in test sets and rated their confidence in their assessment for each case. They also provided a global assessment of their ability to interpret mammograms. Positive predictive value (PPV) and negative predictive value (NPV) were modeled as functions of self-rated confidence on each examination using log-linear regression estimated with generalized estimating equations. Reference measures were cancer status and expert-defined need for recall. Effect modification by weekly mammography volume was examined.
Radiologists who self-reported higher global interpretive ability tended to interpret more mammograms per week (p = 0.08), were more likely to specialize (p = 0.02) and to have completed a fellowship in breast or women's imaging (p = 0.05), and had a higher PPV for cancer detection (p = 0.01). Examinations for which low-volume radiologists were "very confident" had a PPV of 2.93 times (95% CI, 2.01-4.27) higher than examinations they rated with neutral confidence. Trends of increasing NPVs with increasing confidence were significant for low-volume radiologists relative to noncancers (p = 0.01) and expert nonrecalls (p < 0.001). A trend of significantly increasing NPVs existed for high-volume radiologists relative to expert nonrecall (p = 0.02) but not relative to noncancer status (p = 0.32).
Confidence in mammography assessments was associated with better accuracy, especially for low-volume readers. Asking for a second opinion when confidence in an assessment is low may increase accuracy.

Download full-text


Available from: Joann G Elmore, Oct 05, 2015
27 Reads
  • [Show abstract] [Hide abstract]
    ABSTRACT: Physician attributes, job satisfaction and confidence in clinical skills are associated with enhanced performance and better patient outcomes. We surveyed 252 pathologists to evaluate associations between enjoyment of breast pathology, demographic/clinical characteristics and diagnostic performance. Diagnostic performance was determined by comparing pathologist assessments of a set of 60 cases with consensus assessments of the same cases made by a panel of experienced pathologists. Eighty-three percent of study participants reported enjoying breast pathology. Pathologists who enjoy breast interpretation were more likely to review ≥10 cases/week (p = 0.003), report breast interpretation expertise (p = 0.013) and have high levels of confidence interpreting breast pathology (p < 0.001). These pathologists were less likely to report that the field was challenging (p < 0.001) and that breast cases make them more nervous than other types of pathology (p < 0.001). Enjoyment was not associated with diagnostic performance. Millions of women undergo breast biopsy annually, thus it is reassuring that although nearly a fifth of practicing pathologists who interpret breast tissue report not enjoying the field, precision is not impacted. Copyright © 2014 Elsevier Ltd. All rights reserved.
    Breast (Edinburgh, Scotland) 12/2014; 24(2). DOI:10.1016/j.breast.2014.10.003 · 2.38 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: Breast surgeons often see women for second opinions for abnormalities found on breast imaging. For second opinions, these images are submitted for review and interpretation by dedicated breast imagers. This study evaluated the conformity of results among interpretation of imaging submitted from outside hospitals both from tertiary care centers, as well as community programs, in an attempt to evaluate the utility of this practice for the sake of clinical management and resource utilization. A retrospective chart review was conducted on all breast patients that submitted outside imaging films for the years 2011 to 2013 at Rush University Medical Center (RUMC). The radiologic diagnosis and each patient's proposed management plan was collected and evaluated for concordance between the outside institutions and RUMC. A total of 380 patients who presented for second opinions with an interpretation of outside exams were evaluated. In 47.4 % [95 % confidence interval (CI) 42.4-52.4] of cases there was distinct variance in radiologic impression. For 53.5 % (95 % CI 48.4-58.5) of patients, there was a change in recommended management plan, which included recommendations for either additional imaging or need for additional biopsy. In total, this changed the overall surgical management in 27.1 % (95 % CI 22.8-31.9) of cases. In six patients, the reinterpretation of outside imaging detected new malignancies not previously identified. Overall, 83.7 % (95 % CI 79.7-87.1) of patients who submitted imaging from outside institutions chose to complete the remainder of their treatment at RUMC. The practice of second opinion review changed overall definitive management at our specialty center in more than one in four cases. In addition, the review identified six previously unrecognized malignancies. Given this data, the practice of second opinions and interpretation of outside exams should continue despite the additional resources required.
    Annals of Surgical Oncology 01/2015; 22(7). DOI:10.1245/s10434-014-4205-5 · 3.93 Impact Factor
  • [Show abstract] [Hide abstract]
    ABSTRACT: ContextDespite calls for the improvement of self-assessment as a basis for self-directed learning, instructional designs that include reflection in practice are uncommon. Using data from a screen-based simulation for learning radiograph interpretation, we present validity evidence for a simple self-monitoring measure and examine how it can complement skill assessment.Methods Medical students learning ankle radiograph interpretation were given an online learning set of 50 cases which they were asked to classify as ‘abnormal’ (fractured) or ‘normal’ and to indicate the degree to which they felt certain about their response (Definitely or Probably). They received immediate feedback on each case. All students subsequently completed two 20-case post-tests: an immediate post-test (IPT), and a delayed post-test (DPT) administered 2 weeks later. We determined the degree to which certainty (Definitely versus Probably) correlated with accuracy of interpretation and how this relationship changed between the tests.ResultsOf 988 students approached, 115 completed both tests. Mean ± SD accuracy scores decreased from 59 ± 17% at the IPT to 53 ± 16% at the DPT (95% confidence interval [CI] for the difference: −2% to −10%). Mean self-assessed certainty did not decrease (rates of Definitely: IPT, 17.6%; DPT, 19.5%; 95% CI for difference: +7.2% to −3.4%). Regression modelling showed that accuracy was positively associated with choosing Definitely over Probably (odds ratio [OR] 1.63, 95% CI 1.27–2.09) and indicated a statistically significant interaction between test timing and certainty (OR 0.72, 95% CI 0.52–0.99); thus, the accuracy of self-monitoring decayed over the retention interval, leaving students relatively overconfident in their abilities.Conclusions This study shows that, in medical students learning radiograph interpretation, the development of self-monitoring skills can be measured and should not be assumed to necessarily vary in the same way as the underlying clinical skill.
    Medical Education 08/2015; 49(8):838-46. DOI:10.1111/medu.12774 · 3.20 Impact Factor
Show more