Some Practical Issues of Experimental Design and Data Analysis in Radiological ROC Studies

Department of Radiology, University of Chicago, IL 60637.
Investigative Radiology (Impact Factor: 4.45). 04/1989; 24(3):234-45. DOI: 10.1097/00004424-198903000-00012
Source: PubMed

ABSTRACT Receiver operating characteristic (ROC) analysis has been used in a broad variety of medical imaging studies during the past 15 years, and its advantages over more traditional measures of diagnostic performance are now clearly established. But despite the essential simplicity of the approach, workers in the field often find--sometimes only after an ROC study is under way--that a number of subtle issues related to experimental design and data analysis must be confronted in practice. Many of these issues have not been discussed in the literature in detail, and most are not well known. The purposes of this paper are to make users of ROC methodology in medical imaging aware of potential problems that should be confronted before an ROC study is begun and to indicate, at least broadly, how those problems may be dealt with, given the present state of the art. Some of the issues raised here can be addressed adequately by easily prescribed techniques, whereas others remain difficult and will be resolved fully only by new methodologic developments.

  • Source
    • "The key of ROC analysis in radiology is that the participants rate the confidence of judgment or the likelihood of malignancy etc. instead of giving binary answer (i.e. present or absent) (Obuchowski 2003; Metz 1978; Hanley & McNeil 1982; Berbaum et al. 1989; Metz 1989; Gur et al. 1989). The fundamental problem of rating is that the decision needs to be " not obvious " , and " should be of borderline difficulty " (Metz 1978). "
    [Show abstract] [Hide abstract]
    ABSTRACT: Detective performance of radiologists for "obvious" targets should be evaluated by visual search task instead of ROC analysis, but visual task have not been applied to radiology studies. The aim of this study was to set up an environment that allows visual search task in radiology, to evaluate its feasibility, and to preliminarily investigate the effect of career on the performance. In a darkroom, ten radiologists were asked to answer the type of lesion by pressing buttons, when images without lesions, with bulla, ground-glass nodule, and solid nodule were randomly presented on a display. Differences in accuracy and reaction times depending on board certification were investigated. The visual search task was successfully and feasibly performed. Radiologists were found to have high sensitivity, specificity, positive predictive values and negative predictive values in non-board and board groups. Reaction time was under 1 second for all target types in both groups. Board radiologists were significantly faster in answering for bulla, but there were no significant differences for other targets and values. We developed an experimental system that allows visual search experiment in radiology. Reaction time for detection of bulla was shortened with experience.
    SpringerPlus 11/2013; 2:607. DOI:10.1186/2193-1801-2-607
  • Source
    • "In the present study, we evaluated the effectiveness of the QDS using receiver operating characteristic (ROC) analysis [6] [7] [8] [9] [10] [11] [12] from viewpoint of radiation dose reduction. ROC analysis was employed for the objective evaluation of QDS in order to evaluate how far the dose can be reduced with QDS while maintaining an acceptable low-contrast resolution. "
    [Show abstract] [Hide abstract]
    ABSTRACT: For the multislice CT (MSCT) systems with a larger number of detector rows, it is essential to employ dose-reduction techniques. As reported in previous studies, edge-preserving adaptive image filters, which selectively eliminate only the noise elements that are increased when the radiation dose is reduced without affecting the sharpness of images, have been developed. In the present study, we employed receiver operating characteristic (ROC) analysis to assess the effects of the quantum denoising system (QDS), which is an edge-preserving adaptive filter that we have developed, on low-contrast resolution, and to evaluate to what degree the radiation dose can be reduced while maintaining acceptable low-contrast resolution. The low-contrast phantoms (Catphan 412) were scanned at various tube current settings, and ROC analysis was then performed for the groups of images obtained with/without the use of QDS at each tube current to determine whether or not a target could be identified. The tube current settings for which the area under the ROC curve (Az value) was approximately 0.7 were determined for both groups of images with/without the use of QDS. Then, the radiation dose reduction ratio when QDS was used was calculated by converting the determined tube current to the radiation dose. The use of the QDS edge-preserving adaptive image filter allowed the radiation dose to be reduced by up to 38%. The QDS was found to be useful for reducing the radiation dose without affecting the low-contrast resolution in MSCT studies.
    International Journal of Biomedical Imaging 02/2008; 2008:379486. DOI:10.1155/2008/379486
  • Source
    • "The algorithm implemented to detect the clusters of microcalcifications is a five-step process that involves (Lado et al., 1999, 2001): 1) detection of the breast border, employing a tracking algorithm that computes the gradient of grey levels in the breast (Méndez et al., 1996); 2) application of one-dimensional discrete wavelet transform over each column constituting the image; 3) local grey level thresholding to extract the possible microcalcifications , and application of contrast-size test and morphologic operators, to eliminate very small signals, 4) clustering procedure, to group the detected signals; and 5) reduction of false positives, to discriminate between real clusters and false detections, employing several statistical techniques, such as the binormal ROC methodology proposed by Metz (1989). A sensitivity of 80.52% at a false positive rate of 1.90% was achieved. "
    [Show abstract] [Hide abstract]
    ABSTRACT: A local linear method for estimating the conditional ROC curve under the presence of continuous and categorical covariates is introduced. A data driven smoothing parameter selector based on the bootstrap is proposed. The methods are illustrated with real data from a discrimination problem emerging in the context of computer-aided diagnosis. The bootstrap approach is also used to construct pointwise confidence intervals for the area under the ROC curve. (c) 2007 Elsevier B.V. All rights reserved.
    Computational Statistics & Data Analysis 01/2008; 52(5-52):2623-2631. DOI:10.1016/j.csda.2007.09.013 · 1.15 Impact Factor
Show more