Article

StAR: A simple tool for the statistical comparison of ROC curves

Departamento de Genética Molecular y Microbiología, Facultad de Ciencias Biológicas, Pontificia Universidad Católica de Chile, Alameda 340, Santiago, Chile.
BMC Bioinformatics (Impact Factor: 2.58). 02/2008; 9(1):265. DOI: 10.1186/1471-2105-9-265
Source: PubMed

ABSTRACT

As in many different areas of science and technology, most important problems in bioinformatics rely on the proper development and assessment of binary classifiers. A generalized assessment of the performance of binary classifiers is typically carried out through the analysis of their receiver operating characteristic (ROC) curves. The area under the ROC curve (AUC) constitutes a popular indicator of the performance of a binary classifier. However, the assessment of the statistical significance of the difference between any two classifiers based on this measure is not a straightforward task, since not many freely available tools exist. Most existing software is either not free, difficult to use or not easy to automate when a comparative assessment of the performance of many binary classifiers is intended. This constitutes the typical scenario for the optimization of parameters when developing new classifiers and also for their performance validation through the comparison to previous art.
In this work we describe and release new software to assess the statistical significance of the observed difference between the AUCs of any two classifiers for a common task estimated from paired data or unpaired balanced data. The software is able to perform a pairwise comparison of many classifiers in a single run, without requiring any expert or advanced knowledge to use it. The software relies on a non-parametric test for the difference of the AUCs that accounts for the correlation of the ROC curves. The results are displayed graphically and can be easily customized by the user. A human-readable report is generated and the complete data resulting from the analysis are also available for download, which can be used for further analysis with other software. The software is released as a web server that can be used in any client platform and also as a standalone application for the Linux operating system.
A new software for the statistical comparison of ROC curves is released here as a web server and also as standalone software for the LINUX operating system.

Download full-text

Full-text

Available from: Alex William Slater, Jan 10, 2014
  • Source
    • "home.php (accessed 16 October 2014)] (Vergara et al. 2008). Following between-curve comparison, a global ROC curve was computed on the mean scores from the two assessors. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The invasion of non-indigenous freshwater fish species is one of the most important threats to aquatic biodiversity. Similar to other Mediterranean countries, Greece is considered a hot spot for freshwater biodiversity, with many range-restricted endemics of high conservation concern. The aim of this study was to undertake a risk screening assessment to evaluate the invasive potential of non-native, translocated and traded aquarium fishes in Greece by applying the Fish Invasiveness Screening Kit (FISK). In total, 73 freshwater fish species were evaluated by two assessors. FISK was able to discriminate reliably between invasive and non-invasive species with a threshold of 15.25. Based on mean scores, 30 species were classified as ‘high risk’, of which 17 as ‘moderately high risk’, six as ‘high risk’ and seven as ‘very high risk’. There was a high coincidence rate for the species categorisation between the two assessors, but significant differences in certainty. The results suggest that FISK is a useful tool for assessing risks posed by non-native, translocated and traded aquarium fish species in Greece.
    Full-text · Article · Feb 2016 · Fisheries Management and Ecology
  • Source
    • "home.php (accessed 16 October 2014)] (Vergara et al. 2008). Following between-curve comparison, a global ROC curve was computed on the mean scores from the two assessors. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The invasion of non-indigenous freshwater fish species is one of the most important threats to aquatic biodiversity. Similar to other Mediterranean countries, Greece is considered a hot spot for freshwater biodiversity, with many range-restricted endemics of high conservation concern. The aim of the present study was to undertake a risk screening assessment to evaluate the invasive potential of non-native, translocated and traded aquarium fishes in Greece by applying the Fish Invasiveness Screening Kit (FISK). In total, 73 freshwater fish species were evaluated by two assessors. FISK was able to discriminate reliably between invasive and non-invasive species with a threshold of 15.25. Based on mean scores, 30 species were classified as ‘high risk’, of which 17 as ‘moderately high risk’, six as ‘high risk’, and seven as ‘very high risk’. Between the two assessors there was a high coincidence rate for the species categorization, but significant differences in certainty. The results suggest that FISK is a useful tool for assessing risks posed by non-native, translocated and traded aquarium fish species in Greece.
    Full-text · Article · Jul 2015 · Fisheries Management and Ecology
  • Source
    • "home.php) (Vergara et al. 2008). Following betweencurve comparison, a global ROC curve was computed on the mean scores from all 40 species evaluated by the two assessors. "
    [Show abstract] [Hide abstract]
    ABSTRACT: In this first application to Croatia and Slovenia, the Fish Invasiveness Screening Kit (FISK) version 2 was used to assess the invasiveness potential of 40 introduced and translocated freshwater fish species. Based on a priori classification of invasiveness, ROC analysis of FISK scores from two independent assessors resulted in a statistically significant calibration threshold of 11.75. This indicated that FISK was able to discriminate reliably between non-native species likely to pose a high risk of being invasive and those likely to pose a medium or low risk of invasiveness. Seven species were categorised as ‘medium risk’ and the other 33 as ‘high risk’, whereas no species were categorised as ‘low risk’. The two highest scoring species were European catfish and North African catfish. Mean scores for all species classified a priori as invasive were ranked as ‘high risk’ sensu lato and fell into the ‘moderately high risk’ sub-category. FISK proved to be a useful and viable tool for assessing the risks posed by non-native fishes in Croatia and Slovenia. For this reason, it can be adopted as a reliable tool for the prevention of new translocations or introductions of potentially invasive species in the risk assessment area, as well as to assist in decisions regarding future management (i.e. monitoring, control and eradication) and conservation strategies.
    Full-text · Article · Jul 2015 · Fisheries Management and Ecology
Show more