Abstract:Recently, several works have shown that natural modifications of the classical conditional gradient method (aka Frank-Wolfe algorithm) for constrained convex optimization, provably converge with a linear rate when: i) the feasible set is a polytope, and ii) the objective is smooth and strongly-convex. However, all of these results suffer from two significant shortcomings: large memory requirement due to the need to store an explicit convex decomposition of the current iterate, and as a consequence, large running-time overhead per iteration, and worst case convergence rate that depends unfavorably on the dimension. In this work we present a new conditional gradient variant and a corresponding analysis that improves on both of the above shortcomings. In particular: both memory and computation overheads are only linear in the dimension. Moreover, in case the optimal solution is sparse, the new convergence rate replaces a factor which is at least linear in the dimension in previous works, with a linear dependence on the number of non-zeros in the optimal solution. At the heart of our method, and corresponding analysis, is a novel way to compute decomposition-invariant away-steps. While our theoretical guarantees do not apply to any polytope, they apply to several important structured polytopes that capture central concepts such as paths in graphs, perfect matchings in bipartite graphs, marginal distributions that arise in structured prediction tasks, and more. Our theoretical findings are complemented by empirical evidence which shows that our method delivers state-of-the-art performance.

Deconvolution on graphs via linear programming

Blind Deconvolution on Graphs: Exact and Stable Recovery

On-diagonal Lower Estimate of Heat Kernels on Graphs

Volume Doubling, Poincaré Inequality and Gaussian Heat Kernel Estimate for Non-Negatively Curved Graphs

Constructing heat kernels on infinite graphs

Gaussian deconvolution and the lace expansion

Learning Graphs from Linear Measurements: Fundamental Trade-offs and Applications

The parametrix construction of the heat kernel on a graph

Neumann Semigroup, Subgraph Convergence, Form Uniqueness, Stochastic Completeness and the Feller Property

Sharp Davies–Gaffney–Grigor’yan Lemma on Graphs

Heating Up Quasi-Monte Carlo Graph Random Features: A Diffusion Kernel Perspective

Heat kernels on curved cones

Blow-up Phenomenon to the Semilinear Heat Equation for Unbounded Laplacians on Graphs

Linear methods for non-linear inverse problems

Kernels of discrete convolutions and subdivision operators

Ancient Caloric Functions on Graphs with Unbounded Laplacians

A Nonlinear PDE-Based Method for Sparse Deconvolution

Graph-theoretic algorithms for Kolmogorov operators: Approximating solutions and their gradients in elliptic and parabolic problems on manifolds

Linear-memory and Decomposition-invariant Linearly Convergent Conditional Gradient Algorithm for Structured Polytopes

Blind Deconvolution of Sparse Graph Signals in the Presence of Perturbations