Evaluation of Classification Algorithms Using MCDM and Rank Correlation.

Gang Kou,Yanqun Lu,Yi Peng,Yong Shi
DOI: https://doi.org/10.1142/s0219622012500095
2012-01-01
International Journal of Information Technology & Decision Making
Abstract:Classification algorithm selection is an important issue in many disciplines. Since it normally involves more than one criterion, the task of algorithm selection can be modeled as multiple criteria decision making (MCDM) problems. Different MCDM methods evaluate classifiers from different aspects and thus they may produce divergent rankings of classifiers. The goal of this paper is to propose an approach to resolve disagreements among MCDM methods based on Spearman's rank correlation coefficient. Five MCDM methods are examined using 17 classification algorithms and 10 performance criteria over 11 public-domain binary classification datasets in the experimental study. The rankings of classifiers are quite different at first. After applying the proposed approach, the differences among MCDM rankings are largely reduced. The experimental results prove that the proposed approach can resolve conflicting MCDM rankings and reach an agreement among different MCDM methods.
What problem does this paper attempt to address?