Question
Asked 7th Jun, 2013

Is it possible to plot a ROC curve for a multiclass classification algorithm to study its performance, or is it better to analyze by confusion matrix?

I have a a multiclass data-set , which I am analyzing using classification algorithms, but I am having difficultlies plotting the ROC curve. I searched through a lot of papers and sites but most discussions are in relation to binary classification. Is it better to plot a ROC curve for multiclass or just do an analysis of the confusion matrix which could give us a fair idea about the performance of different algorithms?

Most recent answer

9th Feb, 2021
My Hachem Aouragh
Université Moulay Ismail
How?

All Answers (31)

11th Jun, 2013
Jesús Cid-Sueiro
University Carlos III de Madrid
What is a ROC curve in a multiclass problem?
2 Recommendations
11th Jun, 2013
Eric Nunes
Dartmouth College
The best answer could be to treat the multiclass as a binary classification problem that is consider one vs all and calculate the operating points for each class and then average it out for the entire classifier. I am not sure how good of an idea this gives regarding the performance of the classifier, but this is the approach I have seen in many papers.
4 Recommendations
Deleted profile
Sure, you can do that using SPSS and R . You can the roc function in library (pROC) in R. You may see the followiing references:
and
Regards..
12th Sep, 2013
Mahmoud K Okasha
Al-Azhar University - Gaza
Also, it is possible to plot multiclass ROC curve using pROC library in R through the multiclass.roc function.
2 Recommendations
7th Jun, 2015
Dheeb Albashish
Al-Balqa Applied University
May I have a question about the multiclass classification AUC;
if I have three class this means I have also 3 AUC, so my question is which one  can represent my model(system) , or can I take the average of the three AUCs?
Best wishes
7th Jun, 2015
Mahmoud K Okasha
Al-Azhar University - Gaza
Yes Dheeb, you can take the average of the three AUCs. Alternatively, using the levels argument in the multiclass.roc function in pROC library, all levels are used and combined to compute the multiclass AUC. See David J. Hand and Robert J. Till (2001). A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine Learning 45(2), p. 171–186.
Regards 
1 Recommendation
2nd Oct, 2015
André Figueiredo Rendeiro
CeMM Research Center for Molecular Medicine
To complement the answers already here, you might want to have a look at this answer in Stack Overflow.
There are ways of visualizing the performance of multi-label classification tasks other than with ROC curves (e.g. as in attached publication).
4 Recommendations
12th Jun, 2016
Hemant Rana
University College of Medical Sciences
try it multiclass.roc() in R
18th Jan, 2018
Md. Mohaimenul Islam
Taipei Medical University
Weka has this function. You can also try multiclass.roc () function in R
Here is the process-
18th Jan, 2018
Konstantia Barmpatsalou
University of Coimbra
If you use the Orange Machine Learning Software, you will be able to generate multiclass ROCs. However, they will not exist on the same graph.
8th Mar, 2018
Jonathan Q. Li
Vital Scientist, Inc
Thanks people for the interesting and illuminating discussions above. In practice, the situation as follows often comes up:
  • The goal is to identify the best prediction under a competitive multi-class setting, i.e., each class is competing to provide the best answer
    • We allow for the possibility of "no decision".
Suppose Softmax classification is used. How do we come up with a ROC?
10th Mar, 2018
John J Polcari
Oak Ridge National Laboratory
An approach that almost no one uses (because they don't know it), but I consider far superior, would be to compute the consequential information, which is an information measure that captures all the performance issues in a single scalar (and bigger is always better). If you are interested, you can find the development for the binary case in a paper of mine published in IEEE Access, where it is referred to as "separation", and I would be happy to provide the yet to be published notes of how to extend it to the classification problem. The problem with both ROC curves and confusion matrices is they are ambiguous measures of performance; this is not.
16th Apr, 2018
A. Hany Hossny
University of Adelaide
In multi-class classification problem, you either formulate the problem as one-vs-all, where you will have a ROC curve for each class. or you formulate it as one-vs-one, where you will have a ROC curve for all class-pair combinations.
Confusion matrix is nice, but it is not statistically significant as it is a point estimate, except if you will plot the whole Precision-Recall curve and calculate its area.
19th Jul, 2018
Ankush Jamthikar
Visvesvaraya National Institute of Technology
I have a three class problem. My label for the three classes are 0, 1, and 2 for class 1, class 2, and class 3, respectively. I used one-vs-all approach to compute the sensitivities ans specificities. And once I know these two values, I can compute the area under the curve.
But the problem is, since I am using one-vs-all approach, I have three sensitivities, specificities, and AUC. How can I determine the overall values of these parameters for a classifier?
4th Dec, 2018
Matthias Döring
Max Planck Institute for Informatics
I've just written a blog post that discusses plotting ROC curves in the multi-class setting. It is available here:
1 Recommendation
13th Feb, 2019
Yoran Leichsenring
Universidade do Estado de Santa Catarina
What happens in a multi-class "ranking" classifier? I have a classifier that will identify an input of data as one of five available classes by just looking for the higher score (a score is generated for each class). How can I vary a classifier threshold in such a situation? Is it possible?
1 Recommendation
19th Jun, 2019
Amin Ghaderi
You can perform micro or macro averaging of TP and FN ratios across different classes. Sklearn has a great explanation on that.
1 Recommendation
20th Jun, 2019
Mugahed A. Al-antari
Sejong University
- For multi-class problem, you can do vROC (Volume ROC) instead of ROC. Indeed, ROC is just for binary classification task.
- If you want to only use ROC, you can evaluate your model by getting AUC for bi-ROC (between each two-classes) and then average all of them.
- Rather than ROC, there are many evaluation metrics for evaluation such as: Sensitivity, Specificity, Jaccard, F1 score, Overall accuracy, and MCC. You can find the definition of these metrics in my research papers here on ResearchGate.
Thanks!
1 Recommendation
9th Jul, 2019
Saddam Hussain Khan
University of Engineering and Applied Sciences
Both PR and ROC is used for binary classification but if for multiple class, then consider one the positive class and rest of all lies in negative class. so, on this strategy we incorporate mulit class problem.
2 Recommendations
24th Aug, 2019
Don Vicendese
University of Melbourne
Would you be able to point me in the direction where I can learn about vROC please for multi class analysis.
BTW, have you had any experience with the Gerrity score for confusion matrices?
Thank you,
Don
1 Recommendation
4th Sep, 2019
Don Vicendese
University of Melbourne
Thanks very much Seyma
15th Sep, 2019
Mohammed Kemal
Independent researcher
Dear Eric Nunes ,
Great discussion point. Following the updates!
2 Recommendations
3rd Dec, 2020
Viswa Chandu
Sibar Institute of Dental Sciences
Two questions!!
Can we plot multi-class ROC curves in SPSS?
Any insights into sample size requirements for multi-class ROC curves?
Thanks in advance 😊
27th Jan, 2021
My Hachem Aouragh
Université Moulay Ismail
How to plot multiclass.roc like this.
> library(ROCR)
> library(pROC)
> p<-predict(rf_model$finalModel, newdata = testing, type='prob')
> roc.multi<-multiclass.roc(testing$random, p)
Call:
multiclass.roc.default(response = testing$random, predictor = p)
Data: multivariate predictor p with 8 levels of testing$random: Agriculture-Bare soil, Agriculture_Other crops, Arboriculture, Bare Soil, Built-up, Forest, Natural Vegetation, Water Bodies.
Multi-class area under the curve: 0.9755
Thank you
9th Feb, 2021
Yogini Prabhu
Goa University
Offcourse , you can construct a ROC for multi-class classification, as evident as Confusion matrix for it.

Similar questions and discussions

Related Publications

Article
Com o crescente acesso à Web, grande quantidade de conteúdos são produzidos diariamente. O estudo de tais conteúdos permite a descoberta de novos conhecimentos. Nesse sentido, este trabalho apresenta uma análise de algoritmos que permitem a detecção de emoções em tweets no idioma português brasileiro. Assim, são considerados dez algoritmos, desde á...
Article
Full-text available
This paper presents the intrinsic limit determination algorithm (ILD Algorithm), a novel technique to determine the best possible performance, measured in terms of the AUC (area under the ROC curve) and accuracy, that can be obtained from a specific dataset in a binary classification problem with categorical features regardless of the model used. T...
Got a technical question?
Get high-quality answers from experts.