CT colonography with computer-aided detection as a second reader: observer performance study.
ABSTRACT To evaluate the effect of computer-aided detection (CAD) as second reader on radiologists' diagnostic performance in interpreting computed tomographic (CT) colonographic examinations by using a primary two-dimensional (2D) approach, with segmental, unblinded optical colonoscopy as the reference standard.
This HIPAA-compliant study was IRB-approved with written informed consent. Four board-certified radiologists analyzed 60 CT examinations with a commercially available review system. Two-dimensional transverse views were used for initial polyp detection, while three-dimensional (3D) endoluminal and 2D multiplanar views were available for problem solving. After initial review without CAD, the reader was shown CAD-identified polyp candidates. The readers were then allowed to add to or modify their original diagnoses. Polyp location, CT Colonography Reporting and Data System categorization, and reader confidence as to the likelihood of a candidate being a polyp were recorded before and after CAD reading. The area under the receiver operating characteristic (ROC) curve (AUC), sensitivity, and specificity were estimated for CT examinations with and without CAD readings by using multireader multicase analysis.
Use of CAD led to nonsignificant average reader AUC increases of 0.03, 0.03, and 0.04 for patients with adenomatous polyps 6 mm or larger, 6-9 mm, and 10 mm or larger, respectively (P > or = .25); likewise, CAD increased average reader sensitivity by 0.15, 0.16, and 0.14 for those respective groups, with a corresponding decrease in specificity of 0.14. These changes achieved significance for the 6 mm or larger group (P < .01), 6-9 mm group (P < .02), and for specificity (P < .01), but not for the 10 mm or larger group (P > .16). The average reading time was 5.1 minutes +/- 3.4 (standard deviation) without CAD. CAD added an average of 3.1 minutes +/- 4.3 (62%) to each reading (supine and prone positions combined); average total reading time, 8.2 minutes +/- 5.8.
Use of CAD led to a significant increase in sensitivity for detecting polyps in the 6 mm or larger and 6-9 mm groups at the expense of a similar significant reduction in specificity.
- [Show abstract] [Hide abstract]
ABSTRACT: Objectives To assess the effectiveness of computer-aided detection (CAD) as a second reader or concurrent reader in helping radiologists who are moderately experienced in computed tomographic colonography (CTC) to detect colorectal polyps. Methods Seventy CTC datasets (34 patients: 66 polyps ≥6 mm; 36 patients: no abnormalities) were retrospectively reviewed by seven radiologists with moderate CTC experience. After primary unassisted evaluation, a CAD second read and, after a time interval of ≥4 weeks, a CAD concurrent read were performed. Areas under the receiver operating characteristic (ROC) curve (AUC), along with per-segment, per-polyp and per-patient sensitivities, and also reading times, were calculated for each reader with and without CAD. Results Of seven readers, 86 % and 71 % achieved a higher accuracy (segment-level AUC) when using CAD as second and concurrent reader respectively. Average segment-level AUCs with second and concurrent CAD (0.853 and 0.864) were significantly greater (p < 0.0001) than average AUC in the unaided evaluation (0.781). Per-segment, per-polyp, and per-patient sensitivities for polyps ≥6 mm were significantly higher in both CAD reading paradigms compared with unaided evaluation. Second-read CAD reduced readers’ average segment and patient specificity by 0.007 and 0.036 (p = 0.005 and 0.011), respectively. Conclusions CAD significantly improves the sensitivities of radiologists moderately experienced in CTC for polyp detection, both as second reader and concurrent reader. Key Points • CAD helps radiologists with moderate CTC experience to detect polyps ≥6 mm. • Second and concurrent read CAD increase the radiologist’s sensitivity for detecting polyps ≥6 mm. • Second read CAD slightly decreases specificity compared with an unassisted read. • Concurrent read CAD is significantly more time-efficient than second read CAD.European Radiology 07/2014; 24(7). · 4.34 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: Different methods of evaluating diagnostic performance when comparing diagnostic tests may lead to different results. We compared two such approaches, sensitivity and specificity with area under the Receiver Operating Characteristic Curve (ROC AUC) for the evaluation of CT colonography for the detection of polyps, either with or without computer assisted detection.PLoS ONE 10/2014; 9(10):e107633. · 3.53 Impact Factor
- [Show abstract] [Hide abstract]
ABSTRACT: We examined the design, analysis and reporting in multi-reader multi-case (MRMC) research studies using the area under the receiver-operating curve (ROC AUC) as a measure of diagnostic performance. We performed a systematic literature review from 2005 to 2013 inclusive to identify a minimum 50 studies. Articles of diagnostic test accuracy in humans were identified via their citation of key methodological articles dealing with MRMC ROC AUC. Two researchers in consensus then extracted information from primary articles relating to study characteristics and design, methods for reporting study outcomes, model fitting, model assumptions, presentation of results, and interpretation of findings. Results were summarized and presented with a descriptive analysis. Sixty-four full papers were retrieved from 475 identified citations and ultimately 49 articles describing 51 studies were reviewed and extracted. Radiological imaging was the index test in all. Most studies focused on lesion detection vs. characterization and used less than 10 readers. Only 6 (12%) studies trained readers in advance to use the confidence scale used to build the ROC curve. Overall, description of confidence scores, the ROC curve and its analysis was often incomplete. For example, 21 (41%) studies presented no ROC curve and only 3 (6%) described the distribution of confidence scores. Of 30 studies presenting curves, only 4 (13%) presented the data points underlying the curve, thereby allowing assessment of extrapolation. The mean change in AUC was 0.05 (-0.05 to 0.28). Non-significant change in AUC was attributed to underpowering rather than the diagnostic test failing to improve diagnostic accuracy. Data reporting in MRMC studies using ROC AUC as an outcome measure is frequently incomplete, hampering understanding of methods and the reliability of results and study conclusions. Authors using this analysis should be encouraged to provide a full description of their methods and results.PLoS ONE 12/2014; 9(12):e116018. · 3.53 Impact Factor