A Note on Comparison of F-measures

Wei Ju,Wenxin Jiang

DOI: https://doi.org/10.48550/arXiv.2112.04677

IF: 5.414

2021-12-09

Machine Learning

Abstract:We comment on a recent TKDE paper "Linear Approximation of F-measure for the Performance Evaluation of Classification Algorithms on Imbalanced Data Sets", and make two improvements related to comparison of F-measures for two prediction rules.

What problem does this paper attempt to address?

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Xinyue Zhou,Joel R. Levin

DOI: https://doi.org/10.22237/jmasm/1099268100

2004-01-01

Journal of Modern Applied Statistical Methods

Abstract:This article extends Scheffé's modified (sequential) multiple-comparison procedure in one-way analysis-of-variance to other analysis situations, including interaction comparisons in factorial ANOVA designs, tests of partial regression coefficients in multiple-regression analysis, and comparisons of means in one-factor multivariate analyses of variance. Researchers who are concerned with maintaining familywise Type I error rates while increasing statistical power relative to the original (simultaneous) Scheffé-based procedures are encouraged to consider these improved multiple-comparison methods. Copyright © 2004 JMASM, Inc.
Ranking Z-numbers with an improved ranking method for generalized fuzzy numbers.

Wen Jiang,Chunhe Xie,Yu Luo,Yongchuan Tang

DOI: https://doi.org/10.3233/JIFS-16139

2017-01-01

Journal of Intelligent & Fuzzy Systems

Abstract:Z-number, a new concept describes both the restriction and the reliability of evaluation, is more applicable than fuzzy numbers in the fields of decision making, risk assessment etc. However, how to deal with the restriction and the reliability properly is still a problem which is discussed few and inadequately in the existing literatures. In this paper, firstly, a new improved method for ranking generalized fuzzy numbers where the weight of centroid points, degrees of fuzziness and the spreads of fuzzy numbers are taken into consideration is proposed, which can overcome some drawbacks of exiting methods and is very efficient for evaluating symmetric fuzzy numbers and crisp numbers. Then, a procedure for evaluating Znumbers with the method for ranking generalized fuzzy numbers is presented, which considers the status of two parts (((A) over tilde,(B) under tilde)) of Z-numbers and gives the principles of ranking Z-numbers. The main advantage of the proposed method is utilization of Z-numbers which can express more vague information compared with the fuzzy number. In addition, instead of converting (B) over tilde to a crisp number as the existing methods of ranking Z-numbers did, the proposed method retains the fuzzy information of (B) over tilde which can reduce the loss of information. Finally, several numerical examples are provided to illustrate the superiority and the rationality of the proposed procedure.
A Multiple Comparisons Procedure for Detecting Differences Between Treatments and A Control in Two-Factor Experiments

JN Peng,CIC Lee,L Liu

DOI: https://doi.org/10.1080/03610920008832539

2000-01-01

Abstract:In many experiments researchers are interested in comparing several treatment means with a control mean. Their primary interest is to determine whether any treatments are significantly better than the control. Several test procedures have been proposed in the literature, but only few of them can provide simultaneous confidence lower bounds. A new test statistic is proposed to compare treatment means with a control mean in two-factor experiments. Some upper percentage points are tabulated. It yields sharp simultaneous confidence lower bounds for the differences of such means. The new test is unbiased, monotone and consistent. Its power compares favorably with Dunnett's test and the LRT. Also included is an application to scab index data.
A Prediction Method of Fuzzy Association Rules

JJ Lu,BW Xu,JX Jiang

DOI: https://doi.org/10.1109/iri.2003.1251401

2003-01-01

Abstract:Quantitative attributes are partitioned into several fuzzy sets by c-means algorithm, and search technology of Apriori algorithm is improved to discover interesting fuzzy association rules. The first prediction method of fuzzy association rules is presented, and shortcoming of this prediction method is analyzed. Then, the second prediction method of fuzzy association rules with the variable threshold is presented. In the second prediction method, a little error between prediction value and actual value is allowed. When the error is less than a given threshold, prediction value is regarded as acceptable or rational. The second prediction method can obtain the different prediction precision corresponding to the different error threshold chosen by the users, so it is more flexible and effective that the first prediction method.
Application of fuzzy equivalence theory in data cleaning

李华旸,刘玉葆,李又奎

DOI: https://doi.org/10.3969/j.issn.1003-7985.2004.04.012

2004-01-01

Abstract:This paper presents a rule merging and simplifying method and an improved analysis deviation algorithm. The fuzzy equivalence theory avoids the rigid way (either this or that) of traditional equivalence theory. During a data cleaning process task, some rules exist such as "included"/"being included" relations with each other. The equivalence degree of the being-included rule is smaller than that of the including rule, so a rule merging and simplifying method is introduced to reduce the total computing time. And this kind of relation will affect the deviation of fuzzy equivalence degree. An improved analysis deviation algorithm that omits the influence of the included rules' equivalence degree is also presented. Normally the duplicate records are logged in a file, and users have to check and verify them one by one. It's time-cost. The proposed algorithm can save users' labor during duplicate records checking. Finally, an experiment is presented which demonstrates the possibility of the rule.
A Comparative Review of Methods for Comparing Means Using Partially Paired Data

Beibei Guo,Ying Yuan

DOI: https://doi.org/10.1177/0962280215577111

IF: 2.494

2015-01-01

Statistical Methods in Medical Research

Abstract:In medical experiments with the objective of testing the equality of two means, data are often partially paired by design or because of missing data. The partially paired data represent a combination of paired and unpaired observations. In this article, we review and compare nine methods for analyzing partially paired data, including the two-sample t-test, paired t-test, corrected z-test, weighted t-test, pooled t-test, optimal pooled t-test, multiple imputation method, mixed model approach, and the test based on a modified maximum likelihood estimate. We compare the performance of these methods through extensive simulation studies that cover a wide range of scenarios with different effect sizes, sample sizes, and correlations between the paired variables, as well as true underlying distributions. The simulation results suggest that when the sample size is moderate, the test based on the modified maximum likelihood estimator is generally superior to the other approaches when the data is normally distributed and the optimal pooled t-test performs the best when the data is not normally distributed, with well-controlled type I error rates and high statistical power; when the sample size is small, the optimal pooled t-test is to be recommended when both variables have missing data and the paired t-test is to be recommended when only one variable has missing data.
The equivalence theory based on fuzzy theory

Huayang Li,Yubao Liu,Youkui Li,Hao Gui

DOI: https://doi.org/10.1109/ICMLC.2004.1382388

2004-01-01

Abstract:Data cleaning is an important work during the building process of data warehouse and data mining. The equivalence theory means the theory on how to define two records to be equivalent or duplicated. It is an important problem of data cleaning. The paper addressed a new equivalence theory and equivalence degree concept based on fuzzy theory, and put forward the corresponding calculation method of equivalence degrees. Moreover on the basis of the equivalence theory, the key word "report" is introduced and the method of clustering and handling duplicated records is presented. Compared with traditional equivalence theory, the new one is more convenient to generating rules, clustering and handling duplicated records, and reduces user's time of dealing with single LOG files. In addition, the paper put forward an interactive method based on clustering, which saved much of users' labor.
Fuzzy Kappa for the Agreement Measure of Fuzzy Classifications

Weibei Dou,Yuan Ren,Qian Wu,Su Ruan,Yanping Chen,Daniel Bloyet,Jean-Marc Constans

DOI: https://doi.org/10.1016/j.neucom.2006.10.007

IF: 6

2007-01-01

Neurocomputing

Abstract:In this paper, we propose an assessment method of agreement between fuzzy sets, called fuzzy Kappa which is deduced from the concept of Cohen's Kappa statistic. In fuzzy case, the Cohen's Kappa coefficient can be calculated generally by transforming the fuzzy sets into some crisp α-cut subsets. While the proposed fuzzy Kappa allows to directly evaluate an overall agreement between two fuzzy sets. Hence, it is an efficient agreement measure between a given fuzzy “ground truth” or reference and a result of fuzzy classification or fuzzy segmentation. Based on membership function, we define its agreement function and its probability distribution to formulate the deduction of the expectation agreement. So the fuzzy Kappa is calculated from the proportion of the observed agreement and the agreement expected by chance. All the definitions and deductions are detailed in this paper. Both Cohen's Kappa and the fuzzy Kappa are then used to evaluate the agreement between a fuzzy classification of brain tissues on MRI images and its “ground truth”. A comparison of the two types of Kappa coefficient is carried out and shows the advantage of the fuzzy Kappa and some limitations of Cohen's Kappa in the fuzzy case.
Equal Affection or Random Selection: the Quality of Subjective Feedback from a Group Perspective

Jiale Chen,Yuqing Kong,Yuxuan Lu

DOI: https://doi.org/10.48550/arXiv.2102.12247

2021-02-24

Computer Science and Game Theory

Abstract:In the setting where a group of agents is asked a single subjective multi-choice question (e.g. which one do you prefer? cat or dog?), we are interested in evaluating the quality of the collected feedback. However, the collected statistics are not sufficient to reflect how informative the feedback is since fully informative feedback (equal affection of the choices) and fully uninformative feedback (random selection) have the same uniform statistics. Here we distinguish the above two scenarios by additionally asking for respondents' predictions about others' choices. We assume that informative respondents' predictions strongly depend on their own choices while uninformative respondents' do not. With this assumption, we propose a new definition for uninformative feedback and correspondingly design a family of evaluation metrics, called f-variety, for group-level feedback which can 1) distinguish informative feedback and uninformative feedback (separation) even if their statistics are both uniform and 2) decrease as the ratio of uninformative respondents increases (monotonicity). We validate our approach both theoretically and numerically. Moreover, we conduct two real-world case studies about 1) comparisons about athletes and 2) comparisons about stand-up comedians to show the superiority of our approach.
A New Method of Improving the Consistency of Fuzzy Judgement Matrices Based on Compatibility

Hua-you CHEN,Chun-lin LIU

DOI: https://doi.org/10.3969/j.issn.1001-7402.2005.02.024

2005-01-01

Abstract:Abstrcat Fuzzy judgement matrix is a kind of important preference information form provided by (decision) maker. Based on the complementary characteristics of fuzzy judgement matrix, a (compatibility) index is proposed in this paper. The relations between compatibility and consistency of (fuzzy) judgement matrices are also studied. Then a deviation matrix is defined which can reflect the (difference) of fuzzy judgement matrix and its characteristic matrix. A new method of improving the (consistency) of fuzzy judgement matrices is given. Finally, an example shows the feasibility and (effectiveness) of this method.
Testing differences in predictive ability: A tutorial

Tom Fearn

DOI: https://doi.org/10.1002/cem.3549

IF: 2.5

2024-04-27

Journal of Chemometrics

Abstract:This paper describes some statistical tests for comparing the predictive performance of two or more prediction rules. It covers the cases of both quantitative and qualitative predictions, that is, both regression and classification problems. Worked examples are included for both cases.

chemistry, analytical,instruments & instrumentation,mathematics, interdisciplinary applications,automation & control systems,computer science, artificial intelligence,statistics & probability
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

David M. W. Powers

DOI: https://doi.org/10.48550/arXiv.2010.16061

2020-10-11

Abstract:Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic. Using these measures a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance. Informedness and introduce Markedness as a dual measure for the probability that prediction is marked versus chance. Finally we demonstrate elegant connections between the concepts of Informedness, Markedness, Correlation and Significance as well as their intuitive relationships with Recall and Precision, and outline the extension from the dichotomous case to the general multi-class case.

Machine Learning,Methodology
Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

Rob Eisinga,Tom Heskes,Ben Pelzer,Manfred Te Grotenhuis

DOI: https://doi.org/10.1186/s12859-017-1486-2

IF: 3.307

2017-01-25

BMC Bioinformatics

Abstract:BackgroundThe Friedman rank sum test is a widely-used nonparametric method in computational biology. In addition to examining the overall null hypothesis of no significant difference among any of the rank sums, it is typically of interest to conduct pairwise comparison tests. Current approaches to such tests rely on large-sample approximations, due to the numerical complexity of computing the exact distribution. These approximate methods lead to inaccurate estimates in the tail of the distribution, which is most relevant for p-value calculation.ResultsWe propose an efficient, combinatorial exact approach for calculating the probability mass distribution of the rank sum difference statistic for pairwise comparison of Friedman rank sums, and compare exact results with recommended asymptotic approximations. Whereas the chi-squared approximation performs inferiorly to exact computation overall, others, particularly the normal, perform well, except for the extreme tail. Hence exact calculation offers an improvement when small p-values occur following multiple testing correction. Exact inference also enhances the identification of significant differences whenever the observed values are close to the approximate critical value. We illustrate the proposed method in the context of biological machine learning, were Friedman rank sum difference tests are commonly used for the comparison of classifiers over multiple datasets.ConclusionsWe provide a computationally fast method to determine the exact p-value of the absolute rank sum difference of a pair of Friedman rank sums, making asymptotic tests obsolete. Calculation of exact p-values is easy to implement in statistical software and the implementation in R is provided in one of the Additional files and is also available at http://www.ru.nl/publish/pages/726696/friedmanrsd.zip.

biochemical research methods,biotechnology & applied microbiology,mathematical & computational biology
Validation of Overlapping Clustering: A Random Clustering Perspective

Junjie Wu,Hua Yuan,Hui Xiong,Guoqing Chen

DOI: https://doi.org/10.1016/j.ins.2010.07.028

IF: 8.1

2010-01-01

Information Sciences

Abstract:As a widely used clustering validation measure, the F-measure has received increased attention in the field of information retrieval. In this paper, we reveal that the F-measure can lead to biased views as to results of overlapped clusters when it is used for validating the data with different cluster numbers (incremental effect) or different prior probabilities of relevant documents (prior-probability effect). We propose a new “IMplication Intensity” (IMI) measure which is based on the F-measure and is developed from a random clustering perspective. In addition, we carefully investigate the properties of IMI. Finally, experimental results on real-world data sets show that IMI significantly alleviates biased incremental and prior-probability effects which are inherent to the F-measure.
Comparative study of quality estimation of binary classification

V. V. Starovoitov,Yu. I. Golub

DOI: https://doi.org/10.37661/1816-0301-2020-17-1-87-101

2020-03-29

Informatics

Abstract:The paper describes results of analytical and experimental analysis of seventeen functions used for evaluation of binary classification results of arbitrary data. The results are presented by 2×2 error matrices. The behavior and properties of the main functions calculated by the elements of such matrices are studied. Classification options with balanced and imbalanced datasets are analyzed. It is shown that there are linear dependencies between some functions, many functions are invariant to the transposition of the error matrix, which allows us to calculate the estimation without specifying the order in which their elements were written to the matrices. It has been proven that all classical measures such as Sensitivity, Specificity, Precision, Accuracy, F1, F2, GM, the Jacquard index are sensitive to the imbalance of classified data and distort estimation of smaller class objects classification errors. Sensitivity to imbalance is found in the Matthews correlation coefficient and Kohen’s kappa. It has been experimentally shown that functions such as the confusion entropy, the discriminatory power, and the diagnostic odds ratio should not be used for analysis of binary classification of imbalanced datasets. The last two functions are invariant to the imbalance of classified data, but poorly evaluate results with approximately equal common percentage of classification errors in two classes. We proved that the area under the ROC curve (AUC) and the Yuden index calculated from the binary classification confusion matrix are linearly dependent and are the best estimation functions of both balanced and imbalanced datasets.

English Else
An improved sample size calculation method for score tests in generalized linear models

Yongqiang Tang,Liang Zhu,Jiezhun Gu

DOI: https://doi.org/10.48550/arXiv.2006.13104

2020-06-23

Methodology

Abstract:Self and Mauritsen (1988) developed a sample size determination procedure for score tests in generalized linear models under contiguous alternatives. Its performance may deteriorate when the effect size is large. We propose a modification of the Self-Mauritsen method by taking into account of the variance of the score statistic under both the null and alternative hypotheses, and extend the method to noninferiority trials. The modified approach is employed to calculate the sample size for the logistic regression and negative binomial regression in superiority and noninferiority trials. We further explain why the formulae recently derived by Zhu and Lakkis tend to underestimate the required sample size for the negative binomial regression. Numerical examples are used to demonstrate the accuracy of the proposed method.
The MCC-F1 curve: a performance evaluation technique for binary classification

Chang Cao,Davide Chicco,Michael M. Hoffman

DOI: https://doi.org/10.48550/arXiv.2006.11278

IF: 5.414

2020-06-17

Machine Learning

Abstract:Many fields use the ROC curve and the PR curve as standard evaluations of binary classification methods. Analysis of ROC and PR, however, often gives misleading and inflated performance evaluations, especially with an imbalanced ground truth. Here, we demonstrate the problems with ROC and PR analysis through simulations, and propose the MCC-F1 curve to address these drawbacks. The MCC-F1 curve combines two informative single-threshold metrics, MCC and the F1 score. The MCC-F1 curve more clearly differentiates good and bad classifiers, even with imbalanced ground truths. We also introduce the MCC-F1 metric, which provides a single value that integrates many aspects of classifier performance across the whole range of classification thresholds. Finally, we provide an R package that plots MCC-F1 curves and calculates related metrics.
Statistical inference for pairwise comparison models

Ruijian Han,Wenlu Tang,Yiming Xu

2024-04-02

Abstract:Pairwise comparison models have been widely used for utility evaluation and ranking across various fields. The increasing scale of problems today underscores the need to understand statistical inference in these models when the number of subjects diverges, a topic currently lacking in the literature except in a few special instances. To partially address this gap, this paper establishes a near-optimal asymptotic normality result for the maximum likelihood estimator in a broad class of pairwise comparison models, as well as a non-asymptotic convergence rate for each individual subject under comparison. The key idea lies in identifying the Fisher information matrix as a weighted graph Laplacian, which can be studied via a meticulous spectral analysis. Our findings provide a unified theory for performing statistical inference in a wide range of pairwise comparison models beyond the Bradley--Terry model, benefiting practitioners with theoretical guarantees for their use. Simulations utilizing synthetic data are conducted to validate the asymptotic normality result, followed by a hypothesis test using a tennis competition dataset.

Statistics Theory,Machine Learning
Cost-Sensitive Feature Selection by Optimizing F-Measures

Meng Liu,Chang Xu,Yong Luo,Chao Xu,Yonggang Wen,Dacheng Tao

DOI: https://doi.org/10.48550/arXiv.1904.02301

2019-04-04

Computer Vision and Pattern Recognition

Abstract:Feature selection is beneficial for improving the performance of general machine learning tasks by extracting an informative subset from the high-dimensional features. Conventional feature selection methods usually ignore the class imbalance problem, thus the selected features will be biased towards the majority class. Considering that F-measure is a more reasonable performance measure than accuracy for imbalanced data, this paper presents an effective feature selection algorithm that explores the class imbalance issue by optimizing F-measures. Since F-measure optimization can be decomposed into a series of cost-sensitive classification problems, we investigate the cost-sensitive feature selection by generating and assigning different costs to each class with rigorous theory guidance. After solving a series of cost-sensitive feature selection problems, features corresponding to the best F-measure will be selected. In this way, the selected features will fully represent the properties of all classes. Experimental results on popular benchmarks and challenging real-world data sets demonstrate the significance of cost-sensitive feature selection for the imbalanced data setting and validate the effectiveness of the proposed method.
F-measure Maximizing Logistic Regression

Masaaki Okabe,Jun Tsuchida,Hiroshi Yadohisa

DOI: https://doi.org/10.48550/arXiv.1905.02535

2019-05-07

Abstract:Logistic regression is a widely used method in several fields. When applying logistic regression to imbalanced data, for which majority classes dominate over minority classes, all class labels are estimated as `majority class.' In this article, we use an F-measure optimization method to improve the performance of logistic regression applied to imbalanced data. While many F-measure optimization methods adopt a ratio of the estimators to approximate the F-measure, the ratio of the estimators tends to have more bias than when the ratio is directly approximated. Therefore, we employ an approximate F-measure for estimating the relative density ratio. In addition, we define a relative F-measure and approximate the relative F-measure. We show an algorithm for a logistic regression weighted approximated relative to the F-measure. The experimental results using real world data demonstrated that our proposed method is an efficient algorithm to improve the performance of logistic regression applied to imbalanced data.

Methodology,Machine Learning

A Note on Comparison of F-measures

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Ranking Z-numbers with an improved ranking method for generalized fuzzy numbers.

A Multiple Comparisons Procedure for Detecting Differences Between Treatments and A Control in Two-Factor Experiments

A Prediction Method of Fuzzy Association Rules

Application of fuzzy equivalence theory in data cleaning

A Comparative Review of Methods for Comparing Means Using Partially Paired Data

The equivalence theory based on fuzzy theory

Fuzzy Kappa for the Agreement Measure of Fuzzy Classifications

Equal Affection or Random Selection: the Quality of Subjective Feedback from a Group Perspective

A New Method of Improving the Consistency of Fuzzy Judgement Matrices Based on Compatibility

Testing differences in predictive ability: A tutorial

Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers

Validation of Overlapping Clustering: A Random Clustering Perspective

Comparative study of quality estimation of binary classification

An improved sample size calculation method for score tests in generalized linear models

The MCC-F1 curve: a performance evaluation technique for binary classification

Statistical inference for pairwise comparison models

Cost-Sensitive Feature Selection by Optimizing F-Measures

F-measure Maximizing Logistic Regression