Abstract:We examine a new form of smooth approximation to the zero one loss in which learning is performed using a reformulation of the widely used logistic function. Our approach is based on using the posterior mean of a novel generalized Beta-Bernoulli formulation. This leads to a generalized logistic function that approximates the zero one loss, but retains a probabilistic formulation conferring a number of useful properties. The approach is easily generalized to kernel logistic regression and easily integrated into methods for structured prediction. We present experiments in which we learn such models using an optimization method consisting of a combination of gradient descent and coordinate descent using localized grid search so as to escape from local minima. Our experiments indicate that optimization quality is improved when learning metaparameters are themselves optimized using a validation set. Our experiments show improved performance relative to widely used logistic and hinge loss methods on a wide variety of problems ranging from standard UC Irvine and libSVM evaluation datasets to product review predictions and a visual information extraction task. We observe that the approach is as follows: (1) more robust to outliers compared to the logistic and hinge losses; (2) outperforms comparable logistic and max margin models on larger scale benchmark problems; (3) when combined with Gaussian–Laplacian mixture prior on parameters the kernelized version of our formulation yields sparser solutions than Support Vector Machine classifiers; and (4) when integrated into a probabilistic structured prediction technique our approach provides more accurate probabilities yielding improved inference and increasing information extraction performance.

Improved minimax predictive densities under Kullback--Leibler loss

Proper Bayes and Minimax Predictive Densities for a Matrix-variate Normal Distribution

On Bayesian predictive density estimation for skew-normal distributions

Asymptotically minimax Bayesian predictive densities for multinomial models

Bayesian estimation and prediction for certain mixtures

Nearly minimax empirical Bayesian prediction of independent Poisson observables

Understanding and mitigating difficulties in posterior predictive evaluation

Asymptotically Minimax Regret by Bayes Mixtures

Minimaxity under half-Cauchy type priors

Asymptotic Properties of Bayesian Predictive Densities When the Distributions of Data and Target Variables are Different

A PAC-Bayesian Link Between Generalisation and Flat Minima

Bayesian shrinkage prediction for the regression problem

High-probability minimax lower bounds

Adaptive Bayesian density estimation using Pitman-Yor or normalized inverse-Gaussian process kernel mixtures

A Few Interactions Improve Distributed Nonparametric Estimation, Optimally

Adaptive Bayesian multivariate density estimation with Dirichlet mixtures

Minimizing Convex Functionals over Space of Probability Measures via KL Divergence Gradient Flow

Generalized multi-view model: Adaptive density estimation under low-rank constraints

A New Smooth Approximation to the Zero One Loss with a Probabilistic Interpretation

Asymptotics of predictive distributions driven by sample means and variances