Abstract:We propose a family of recursive cutting-plane algorithms to solve feasibility problems with constrained memory, which can also be used for first-order convex optimization. Precisely, in order to find a point within a ball of radius $\epsilon$ with a separation oracle in dimension $d$ -- or to minimize $1$-Lipschitz convex functions to accuracy $\epsilon$ over the unit ball -- our algorithms use $\mathcal O(\frac{d^2}{p}\ln \frac{1}{\epsilon})$ bits of memory, and make $\mathcal O((C\frac{d}{p}\ln \frac{1}{\epsilon})^p)$ oracle calls, for some universal constant $C \geq 1$. The family is parametrized by $p\in[d]$ and provides an oracle-complexity/memory trade-off in the sub-polynomial regime $\ln\frac{1}{\epsilon}\gg\ln d$. While several works gave lower-bound trade-offs (impossibility results) -- we explicit here their dependence with $\ln\frac{1}{\epsilon}$, showing that these also hold in any sub-polynomial regime -- to the best of our knowledge this is the first class of algorithms that provides a positive trade-off between gradient descent and cutting-plane methods in any regime with $\epsilon\leq 1/\sqrt d$. The algorithms divide the $d$ variables into $p$ blocks and optimize over blocks sequentially, with approximate separation vectors constructed using a variant of Vaidya's method. In the regime $\epsilon \leq d^{-\Omega(d)}$, our algorithm with $p=d$ achieves the information-theoretic optimal memory usage and improves the oracle-complexity of gradient descent.

An optimal variant of Kelley’s cutting-plane method

A Cutting Plane and Level Stabilization Bundle Method with Inexact Data for Minimizing Nonsmooth Nonconvex Functions

Norm-induced Cuts: Optimization with Lipschitzian Black-box Functions

A cutting-plane method to nonsmooth multiobjective optimization problems

Accelerated method of finding for the minimum of arbitrary Lipschitz convex function

Yet another fast variant of Newton’s method for nonconvex optimization

Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints

An entropy based central cutting plane algorithm for convex min-max semi-infinite programming problems

PolyCD: Optimization via Cycling through the Vertices of a Polytope

Exact Lipschitz Regularization of Convex Optimization Problems

Memory-Constrained Algorithms for Convex Optimization via Recursive Cutting-Planes

An Infeasible-Point Subgradient Method Using Adaptive Approximate Projections

Curves of Minimax Curvature

Adaptive Algorithms for Relatively Lipschitz Continuous Convex Optimization Problems

An interior proximal gradient method for nonconvex optimization

First Order Methods beyond Convexity and Lipschitz Gradient Continuity with Applications to Quadratic Inverse Problems

Coordinate descent methods beyond smoothness and separability

Lipschitz minimization and the Goldstein modulus

Stochastic Cutting Planes for Data-Driven Optimization

An implementable proximal-type method for computing critical points to minimization problems with a nonsmooth and nonconvex constraint