On the use of the overlapping area matrix for image segmentation evaluation: A survey and new performance measures

Department of Mathematic and Computer Sciences , University of the Balearic Islands, Palma, Balearic Islands, Spain
Pattern Recognition Letters (Impact Factor: 1.55). 01/2006; DOI: 10.1016/j.patrec.2006.05.002
Source: DBLP


The development of common and reasonable criteria for evaluating and comparing the performance of segmentation algorithms has always been a concern for researchers in the area. As it is discussed in the paper, some of the measures proposed are not adequate for general images (i.e. images of any sort of scene, without any assumption about the features of the scene objects or the illumination distribution) because they assume a certain distribution of pixel gray-level or colour values for the interior of the regions. This paper reviews performance measures not performing such an assumption and proposes a set of new performance measures in the same line, called the percentage of correctly grouped pixels (CG), the percentage of over-segmentation (OS) and the percentage of under-segmentation (US). Apart from accounting for misclassified pixels, the proposed set of new measures are intended to compute the level of fragmentation of reference regions into output regions and vice versa. A comparison involving similar measures is provided at the end of the paper.

21 Reads
    • "Thereby , less than 1% of the agricultural fields in the study sites were discarded from the classification . After segmentation , an empirical discrepancy method was applied to measure the quality of the segmented objects ( Ortiz and Oliver , 2006 ; Zhang , 1996 ) . This was accomplished by assess - ing the similarity of the outputs from the image segmentation with 400 randomly distributed reference polygons that were digitized on - screen . "
    [Show abstract] [Hide abstract]
    ABSTRACT: Agricultural management increasingly uses crop maps based on classification of remotely sensed data. However, classification errors can translate to errors in model outputs, for instance agricultural production monitoring (yield, water demand) or crop acreage calculation. Hence, knowledge on the spatial variability of the classier performance is important information for the user. But this is not provided by traditional assessments of accuracy, which are based on the confusion matrix. In this study, classification uncertainty was analyzed, based on the support vector machines (SVM) algorithm. SVM was applied to multi-spectral time series data of RapidEye from different agricultural landscapes and years. Entropy was calculated as a measure of classification uncertainty, based on the per-object class membership estimations from the SVM algorithm. Permuting all possible combinations of available images allowed investigating the impact of the image acquisition frequency and timing, respectively, on the classification uncertainty. Results show that multi-temporal datasets decrease classification uncertainty for different crops compared to single data sets, but there was no “one-image-combination-fits-all” solution. The number and acquisition timing of the images, for which a decrease in uncertainty could be realized, proved to be specific to a given landscape, and for each crop they differed across different landscapes. For some crops, an increase of uncertainty was observed when increasing the quantity of images, even if classification accuracy was improved. Random forest regression was employed to investigate the impact of different explanatory variables on the observed spatial pattern of classification uncertainty. It was strongly influenced by factors related with the agricultural management and training sample density. Lower uncertainties were revealed for fields close to rivers or irrigation canals. This study demonstrates that classification uncertainty estimates by the SVM algorithm provide a valuable addition to traditional accuracy assessments. This allows analyzing spatial variations of the classifier performance in maps and also differences in classification uncertainty within the growing season and between crop types, respectively.
    ISPRS Journal of Photogrammetry and Remote Sensing 07/2015; 105. DOI:10.1016/j.isprsjprs.2015.03.004 · 3.13 Impact Factor
  • Source
    • "Given all performance measures that can be based on pixel counts or object-based detection rates, a final task of interest is to rank the detection algorithms according to their overall performance . Most of the studies (Huang and Dom, 1995; Hoover et al., 1996; Mariano et al., 2002; Ortiz and Oliver, 2006; Jiang "
    [Show abstract] [Hide abstract]
    ABSTRACT: We propose a new procedure for quantitative evaluation of object detection algorithms. The procedure consists of a matching stage for finding correspondences between reference and output objects, an accuracy score that is sensitive to object shapes as well as boundary and fragmentation errors, and a ranking step for final ordering of the algorithms using multiple performance indicators. The procedure is illustrated on a building detection task where the resulting rankings are consistent with the visual inspection of the detection maps.
    Pattern Recognition Letters 07/2010; 31(10):1128-1137. DOI:10.1016/j.patrec.2009.10.016 · 1.55 Impact Factor
  • Source
    • "(black or white) and two phases are defined (Ortiz and Oliver, 2006; 123 Seul et al., 2006; Gonzalez and Woods, 1992; Russ, 1998; Bernd, 124 1997). In this case, binary segmentation was fixed to the threshold "
    [Show abstract] [Hide abstract]
    ABSTRACT: In this research the spatial distribution characterization of niobium phosphate into bleached cellulose was carried out combining processing and images analysis obtained by SEM and statistical methodologies. The objective is to investigate the deposit composition and phosphate morphology by using complementary analytical techniques. Based on the proposed methodology, parameters of niobium phosphate agglomerates (size and shape) and fiber morphology were evaluated depending on gray-levels (average luminance and fiber type): fiber characteristics (morphology) were measured. For the test method proposed, a specific region of cellulose/NbOPO(4) x nH(2)O composite was analyzed. This method involves area fraction measuring with a conditional probabilistic analysis. The analyzed fields were divided in different ways, called 'Scanning' and as a result, in quantitative terms, the phosphate deposition was described as spatial distribution homogeneous or inhomogeneous. The quantitative microscopy as a non-destructive testing provides relevant information when it is combined with statistic analysis.
    Micron 03/2010; 41(5):402-11. DOI:10.1016/j.micron.2010.02.012 · 1.99 Impact Factor
Show more