Abstract:We revisit the sequential variants of linear regression with the squared loss, classification problems with hinge loss, and logistic regression, all characterized by unbounded losses in the setup where no assumptions are made on the magnitude of design vectors and the norm of the optimal vector of parameters. The key distinction from existing results lies in our assumption that the set of design vectors is known in advance (though their order is not), a setup sometimes referred to as transductive online learning. While this assumption seems similar to fixed design regression or denoising, we demonstrate that the sequential nature of our algorithms allows us to convert our bounds into statistical ones with random design without making any additional assumptions about the distribution of the design vectors--an impossibility for standard denoising results. Our key tools are based on the exponential weights algorithm with carefully chosen transductive (design-dependent) priors, which exploit the full horizon of the design vectors. Our classification regret bounds have a feature that is only attributed to bounded losses in the literature: they depend solely on the dimension of the parameter space and on the number of rounds, independent of the design vectors or the norm of the optimal solution. For linear regression with squared loss, we further extend our analysis to the sparse case, providing sparsity regret bounds that additionally depend on the magnitude of the response variables. We argue that these improved bounds are specific to the transductive setting and unattainable in the worst-case sequential setup. Our algorithms, in several cases, have polynomial time approximations and reduce to sampling with respect to log-concave measures instead of aggregating over hard-to-construct $\varepsilon$-covers of classes.

Sequential Linearithmic Time Optimal Unimodal Fitting When Minimizing Univariate Linear Losses

Optimally Efficient Sequential Calibration of Binary Classifiers to Minimize Classification Error

A Log-Linear Time Sequential Optimal Calibration Algorithm for Quantized Isotonic L2 Regression

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

Optimal convex $M$-estimation via score matching

Sequential Labeling with Structural Svm under the F-1 Loss

A Non-linear Function-on-Function Model for Regression with Time Series Data

A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models

Sequential estimation for the multiple linear regression models with balanced loss functions

Online Efficient Learning with Quantized Klms and L-1 Regularization

Online Efficient Learning with Quantized KLMS and L<inf>1</inf> Regularization

Optimal Subsampling for Large Sample Logistic Regression

Nonlinear Estimation Based on Conversion-Sample Optimization

Sublinear Dual Coordinate Ascent for Regularized Loss Minimization.

Sequential Labeling with Structural SVM under Nondecomposable Losses.

Predicting Nearly As Well As the Optimal Twice Differentiable Regressor

GORFLM: Globally Optimal Robust Fitting for Linear Model

Single Sequential Minimal Optimization: an Improved SVMs Training Algorithm

Sequential minimal optimization for SVM with pinball loss.

Minimax Optimal Kernel Operator Learning via Multilevel Training

Refined Risk Bounds for Unbounded Losses via Transductive Priors