Abstract:We develop a variant of multiclass logistic regression that is significantly more robust to noise. The algorithm has one weight vector per class and the surrogate loss is a function of the linear activations (one per class). The surrogate loss of an example with linear activation vector $\mathbf{a}$ and class $c$ has the form $-\log_{t_1} \exp_{t_2} (a_c - G_{t_2}(\mathbf{a}))$ where the two temperatures $t_1$ and $t_2$ ''temper'' the $\log$ and $\exp$, respectively, and $G_{t_2}(\mathbf{a})$ is a scalar value that generalizes the log-partition function. We motivate this loss using the Tsallis divergence. Our method allows transitioning between non-convex and convex losses by the choice of the temperature parameters. As the temperature $t_1$ of the logarithm becomes smaller than the temperature $t_2$ of the exponential, the surrogate loss becomes ''quasi convex''. Various tunings of the temperatures recover previous methods and tuning the degree of non-convexity is crucial in the experiments. In particular, quasi-convexity and boundedness of the loss provide significant robustness to the outliers. We explain this by showing that $t_1 < 1$ caps the surrogate loss and $t_2 >1$ makes the predictive distribution have a heavy tail. We show that the surrogate loss is Bayes-consistent, even in the non-convex case. Additionally, we provide efficient iterative algorithms for calculating the log-partition value only in a few number of iterations. Our compelling experimental results on large real-world datasets show the advantage of using the two-temperature variant in the noisy as well as the noise free case.

Generalization of the Kullback–Leibler Divergence in the Tsallis Statistics

Detecting abnormal situations using the Kullback-Leibler divergence.

The $q$-generalised multinomial/KL-divergence correspondence

Process Monitoring Based on Kullback Leibler Divergence

Central Limit Theorem on Symmetric Kullback-Leibler (KL) Divergence

Tsallis entropy and generalized Shannon additivity

Tsallis eXtropy

Exact Expressions for Kullback–Leibler Divergence for Univariate Distributions

Sup-sums principles for F-divergence, Kullback--Leibler divergence, and new definition for t-entropy

A new expression of the probability distribution in incomplete statistics and fundamental thermodynamic relations

Taming numerical imprecision by adapting the KL divergence to negative probabilities

Tsallis thermostatics as a statistical physics of random chains

Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergence

Transport information Bregman divergences

Axioms and uniqueness theorem for Tsallis entropy

Two-temperature logistic regression based on the Tsallis divergence

Convergence rate of Tsallis entropic regularized optimal transport

Trivergence of Probability Distributions, at glance

On the exact variance of Tsallis entropy in a random pure state

On Tsallis extropy with an application to pattern recognition

On the Second-Order Asymptotics of the Hoeffding Test and Other Divergence Tests