Abstract:Dropout and other feature noising schemes have shown promising results in controlling over-fitting by artificially corrupting the training data. Though extensive theoretical and empirical studies have been performed for generalized linear models, little work has been done for support vector machines (SVMs), one of the most successful approaches for supervised learning. This paper presents dropout training for both linear SVMs and the nonlinear extension with latent representation learning. For linear SVMs, to deal with the intractable expectation of the non-smooth hinge loss under corrupting distributions, we develop an iteratively re-weighted least square (IRLS) algorithm by exploring data augmentation techniques. Our algorithm iteratively minimizes the expectation of a re-weighted least square problem, where the re-weights are analytically updated. For nonlinear latent SVMs, we consider learning one layer of latent representations in SVMs and extend the data augmentation technique in conjunction with first-order Taylor-expansion to deal with the intractable expected non-smooth hinge loss and the nonlinearity of latent representations. Finally, we apply the similar data augmentation ideas to develop a new IRLS algorithm for the expected logistic loss under corrupting distributions, and we further develop a non-linear extension of logistic regression by incorporating one layer of latent representations. Our algorithms offer insights on the connection and difference between the hinge loss and logistic loss in dropout training. Empirical results on several real datasets demonstrate the effectiveness of dropout training on significantly boosting the classification accuracy of both linear and nonlinear SVMs. In addition, the nonlinear SVMs further improve the prediction performance on several image datasets.

Dropout Training for Support Vector Machines

Dropout Training for SVMs with Data Augmentation

Wordreg: Mitigating the Gap Between Training and Inference with Worst-Case Drop Regularization

An Incremental Updating Method for Support Vector Machines

Incremental batch learning with support vector machines

M-estimator Based Support Vector Machine and Its Application

Robust support vector machine training via convex outlier ablation

Dropout Reduces Underfitting

Adaptive Dropout Rates for Learning with Corrupted Features

R-Drop: Regularized Dropout for Neural Networks.

Dropout, a basic and effective regularization method for a deep learning model: a case study

Robust Support Vector Data Description with Truncated Loss Function for Outliers Depression

<inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math></inline-formula>-Dropout: A Unified Dropout

Data Dropout: Optimizing Training Data for Convolutional Neural Networks

Convex large margin training techniques: unsupervised, semi-supervised, and robust support vector machines

Robust Support Vector Machine Using Least Median Loss Penalty

Stochastic Modified Equations and Dynamics of Dropout Algorithm

&Lt;inline-Formula> &Lt;tex-Math Notation="latex">$\beta$ &Lt;/tex-Math> &Lt;/inline-Formula>-dropout: A Unified Dropout

Functional iterative approaches for solving support vector classification problems based on generalized Huber loss

Robust multiclass least squares support vector classifier with optimal error distribution

<inline-formula> <tex-math notation="LaTeX">$\beta$ </tex-math> </inline-formula>-Dropout: A Unified Dropout