Abstract:Gradient-based minimax optimal algorithms have greatly promoted the development of continuous optimization and machine learning. One seminal work due to Yurii Nesterov [Nes83a] established $\tilde{\mathcal{O}}(\sqrt{L/\mu})$ gradient complexity for minimizing an $L$-smooth $\mu$-strongly convex objective. However, an ideal algorithm would adapt to the explicit complexity of a particular objective function and incur faster rates for simpler problems, triggering our reconsideration of two defeats of existing optimization modeling and analysis. (i) The worst-case optimality is neither the instance optimality nor such one in reality. (ii) Traditional $L$-smoothness condition may not be the primary abstraction/characterization for modern practical problems. In this paper, we open up a new way to design and analyze gradient-based algorithms with direct applications in machine learning, including linear regression and beyond. We introduce two factors $(\alpha, \tau_{\alpha})$ to refine the description of the degenerated condition of the optimization problems based on the observation that the singular values of Hessian often drop sharply. We design adaptive algorithms that solve simpler problems without pre-known knowledge with reduced gradient or analogous oracle accesses. The algorithms also improve the state-of-art complexities for several problems in machine learning, thereby solving the open problem of how to design faster algorithms in light of the known complexity lower bounds. Specially, with the $\mathcal{O}(1)$-nuclear norm bounded, we achieve an optimal $\tilde{\mathcal{O}}(\mu^{-1/3})$ (v.s. $\tilde{\mathcal{O}}(\mu^{-1/2})$) gradient complexity for linear regression. We hope this work could invoke the rethinking for understanding the difficulty of modern problems in optimization.

Gradient Learning With the Mode-Induced Loss: Consistency Analysis and Applications

Expectile regression for analyzing heteroscedasticity in high dimension

Gradient Learning Approach For Variable Selection In Credit Scoring

Variable Selection in Convex Piecewise Linear Regression

CGLearn: Consistent Gradient-Based Learning for Out-of-Distribution Generalization

Sensitivity Analysis On Loss Landscape

Directional Smoothness and Gradient Methods: Convergence and Adaptivity

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Gradient-Aware Logit Adjustment Loss for Long-tailed Classifier

The column measure and Gradient-Free Gradient Boosting

Accelerated gradient methods for sparse statistical learning with nonconvex penalties

Gradient-Variation Online Learning under Generalized Smoothness

Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minima

A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models

Gradient-based sparse principal component analysis with extensions to online learning

Gradient-induced variable selection in reproducing kernel Hilbert space for survival analysis

Beyond the Edge of Stability via Two-step Gradient Updates

Nonconvex Sparse Logistic Regression Via Proximal Gradient Descent.

Variable Selection for High-dimensional Generalized Linear Models using an Iterated Conditional Modes/Medians Algorithm

Classifier-guided Gradient Modulation for Enhanced Multimodal Learning