Abstract:An algorithm is said to be adaptive to a certain parameter (of the problem) if it does not need a priori knowledge of such a parameter but performs competitively to those that know it. This dissertation presents our work on adaptive algorithms in following scenarios: 1. In the stochastic optimization setting, we only receive stochastic gradients and the level of noise in evaluating them greatly affects the convergence rate. Tuning is typically required when without prior knowledge of the noise scale in order to achieve the optimal rate. Considering this, we designed and analyzed noise-adaptive algorithms that can automatically ensure (near)-optimal rates under different noise scales without knowing it. 2. In training deep neural networks, the scales of gradient magnitudes in each coordinate can scatter across a very wide range unless normalization techniques, like BatchNorm, are employed. In such situations, algorithms not addressing this problem of gradient scales can behave very poorly. To mitigate this, we formally established the advantage of scale-free algorithms that adapt to the gradient scales and presented its real benefits in empirical experiments. 3. Traditional analyses in non-convex optimization typically rely on the smoothness assumption. Yet, this condition does not capture the properties of some deep learning objective functions, including the ones involving Long Short-Term Memory networks and Transformers. Instead, they satisfy a much more relaxed condition, with potentially unbounded smoothness. Under this condition, we show that a generalized SignSGD algorithm can theoretically match the best-known convergence rates obtained by SGD with gradient clipping but does not need explicit clipping at all, and it can empirically match the performance of Adam and beat others. Moreover, it can also be made to automatically adapt to the unknown relaxed smoothness.

Adaptive sieving: A dimension reduction technique for sparse optimization problems

A Dimension Reduction Technique for Large-scale Structured Sparse Optimization Problems with Application to Convex Clustering

An Overview of Computational Sparse Models and Their Applications in Artificial Intelligence

An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression

Survey of Solving the Optimization Problems for Sparse Learning

Adaptive Partitioning Strategy for High-Dimensional Discrete Simulation-based Optimization Problems

An Efficient Sieving-Based Secant Method for Sparse Optimization Problems with Least-Squares Constraints

An Efficient Algorithm for a Class of Large-Scale Support Vector Machines Exploiting Hidden Sparsity.

Support matrix machine: exploring sample sparsity, low rank, and adaptive sieving in high-performance computing

An efficient sieving based secant method for sparse optimization problems with least-squares constraints

SPARSE SIR: OPTIMAL RATES AND ADAPTIVE ESTIMATION

Solving Stochastic Optimization by Newton-type methods with Dimension-Adaptive Sparse Grid Quadrature

Group sparse sufficient dimension reduction: a model-free group variable selection method

A Unified Dynamic Approach to Sparse Model Selection

Adaptive Strategies in Non-convex Optimization

Adaptive Strategies For Efficient Model Reduction In High-Dimensional Inverse Problems

Structured Sparsity Optimization with Non-Convex Surrogates of L2,0-Norm: A Unified Algorithmic Framework

A Survey on Intelligent Iterative Methods for Solving Sparse Linear Algebraic Equations

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Randomized Dimension Reduction with Statistical Guarantees

A sparse semismooth Newton based augmented Lagrangian method for large-scale support vector machines