Abstract:We consider the most common variants of linear regression, including Ridge, Lasso and Support-vector regression, in a setting where the learner is allowed to observe only a fixed number of attributes of each example at training time. We present simple and efficient algorithms for these problems: for Lasso and Ridge regression they need the same total number of attributes (up to constants) as do full-information algorithms, for reaching a certain accuracy. For Support-vector regression, we require exponentially less attributes compared to the state of the art. By that, we resolve an open problem recently posed by Cesa-Bianchi et al. (2010). Experiments show the theoretical bounds to be justified by superior performance compared to the state of the art.

What problem does this paper attempt to address?

This paper attempts to address the problem of performing linear regression under the Limited Attribute Observation (LAO) setting. Specifically, the researchers focus on how to effectively conduct Ridge Regression, Lasso Regression, and Support Vector Regression (SVR) when only a small portion of attributes for each sample can be observed during training. The main issues can be summarized as follows: 1. **Can the same accuracy be achieved with the same total number of attributes under the LAO setting as in the unrestricted case?** - The paper demonstrates through the proposal of efficient algorithms that for Ridge Regression and Lasso Regression, the same accuracy can be achieved with the same total number of attributes under the LAO setting as in the unrestricted case. - For Support Vector Regression, the paper proposes significantly improved methods that reduce parameter dependency exponentially. 2. **How to design effective algorithms to handle linear regression problems under the LAO setting?** - The paper proposes several simple and efficient algorithms that perform well in experiments, outperforming existing methods. - Specifically, for Ridge Regression and Lasso Regression, the paper proposes algorithms based on Randomized Online Gradient Descent (OGD) and Multiplicative Weights Update (MWU). - For Support Vector Regression, the paper proposes algorithms based on biased gradient estimators and addresses the issue of non-smooth loss functions through approximate loss functions. 3. **What is the impact of Limited Attribute Observation on regression learning?** - Through theoretical analysis and experimental validation, the paper shows that regression learning performance can still remain high under the LAO setting, especially when the sample size is sufficiently large. - The paper also demonstrates that in some cases, LAO does not significantly affect learning efficiency, providing a theoretical basis for resource optimization in practical applications. In summary, this paper addresses key issues in performing linear regression under the LAO setting by proposing new algorithms and theoretical analyses, providing important references for research and applications in related fields.

Linear Regression with Limited Observation

Electronic structure study by means of X-ray spectroscopy and theoretical calculations of the "ferric star" single molecule magnet

Bounded Support In Linear Random Coefficient Models: Identification And Variable Selection

Asymptotics of Linear Regression with Linearly Dependent Data

On Ridge Estimation in High-dimensional Rotationally Sparse Linear Regression

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

Asymptotic Properties of Lasso+mLS and Lasso+Ridge in Sparse High-Dimensional Linear Regression

Linear and Conic Programming Estimators in High-Dimensional Errors-in-variables Models

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Sparse Quadratic Logistic Regression in Sub-quadratic Time

To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets

Lasso with Latents: Efficient Estimation, Covariate Rescaling, and Computational-Statistical Gaps

Conditional Sparse Linear Regression

Robust Satisficing Linear Regression: performance/robustness trade-off and consistency criterion

No penalty no tears: Least squares in high-dimensional linear models

Sparse and Robust Linear Regression: An Optimization Algorithm and Its Statistical Properties

A Study of Error Variance Estimation in Lasso Regression

On low-rank regularized least squares for scalable nonlinear classification

An algebraic-geometric approach for linear regression without correspondences

Regression from Dependent Observations