Abstract:Recently, several works have shown that natural modifications of the classical conditional gradient method (aka Frank-Wolfe algorithm) for constrained convex optimization, provably converge with a linear rate when: i) the feasible set is a polytope, and ii) the objective is smooth and strongly-convex. However, all of these results suffer from two significant shortcomings: large memory requirement due to the need to store an explicit convex decomposition of the current iterate, and as a consequence, large running-time overhead per iteration, and worst case convergence rate that depends unfavorably on the dimension. In this work we present a new conditional gradient variant and a corresponding analysis that improves on both of the above shortcomings. In particular: both memory and computation overheads are only linear in the dimension. Moreover, in case the optimal solution is sparse, the new convergence rate replaces a factor which is at least linear in the dimension in previous works, with a linear dependence on the number of non-zeros in the optimal solution. At the heart of our method, and corresponding analysis, is a novel way to compute decomposition-invariant away-steps. While our theoretical guarantees do not apply to any polytope, they apply to several important structured polytopes that capture central concepts such as paths in graphs, perfect matchings in bipartite graphs, marginal distributions that arise in structured prediction tasks, and more. Our theoretical findings are complemented by empirical evidence which shows that our method delivers state-of-the-art performance.

A geodesic convexity-like structure for the polar decomposition of a square matrix

Making matrices better: Geometry and topology of polar and singular value decomposition

Linear-memory and Decomposition-invariant Linearly Convergent Conditional Gradient Algorithm for Structured Polytopes

The polarization hierarchy for polynomial optimization over convex bodies, with applications to nonnegative matrix rank

On orthogonal polar spaces

A Backward Stable Algorithm for Computing the CS Decomposition via the Polar Decomposition

A topological insight into the polar involution of convex sets

Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Steepest Descent

On SVD and Polar Decomposition in Real and Complexified Clifford Algebras

General Algorithms for the Polar Decomposition and Strains

Association schemes and orthogonality graphs on anisotropic points of polar spaces

A Feasible Method for Optimization with Orthogonality Constraints

Sum-of-squares Chordal Decomposition of Polynomial Matrix Inequalities

On the Geometric Analysis of a Quartic–quadratic Optimization Problem under a Spherical Constraint

Efficient coordinate-descent for orthogonal matrices through Givens rotations

A Bicomposition of Conical Projections

Symmetry & Critical Points for Symmetric Tensor Decomposition Problems

Geodesic Convexity of the Symmetric Eigenvalue Problem and Convergence of Riemannian Steepest Descent

Polar duality and the reconstruction of quantum covariance matrices from partial data