Abstract:A vast majority of machine learning algorithms train their models and perform inference by solving optimization problems. In order to capture the learning and prediction problems accurately, structural constraints such as sparsity or low rank are frequently imposed or else the objective itself is designed to be a non-convex function. This is especially true of algorithms that operate in high-dimensional spaces or that train non-linear models such as tensor models and deep networks. The freedom to express the learning problem as a non-convex optimization problem gives immense modeling power to the algorithm designer, but often such problems are NP-hard to solve. A popular workaround to this has been to relax non-convex problems to convex ones and use traditional methods to solve the (convex) relaxed optimization problems. However this approach may be lossy and nevertheless presents significant challenges for large scale optimization. On the other hand, direct approaches to non-convex optimization have met with resounding success in several domains and remain the methods of choice for the practitioner, as they frequently outperform relaxation-based techniques – popular heuristics include projected gradient descent and alternating minimization. However, these are often poorly understood in terms of their convergence and other properties. This monograph presents a selection of recent advances that bridge a long-standing gap in our understanding of these heuristics. We hope that an insight into the inner workings of these methods will allow the reader to appreciate the unique marriage of task structure and generative models that allow these heuristic techniques to (provably) succeed. The monograph will lead the reader through several widely used nonconvex optimization techniques, as well as applications thereof. The goal of this monograph is to both, introduce the rich literature in this area, as well as equip the reader with the tools and techniques needed to analyze these simple procedures for non-convex problems.

Recent Advances in Non-convex Smoothness Conditions and Applicability to Deep Linear Neural Networks

Extended convexity and smoothness and their applications in deep learning

Convex and Non-convex Optimization Under Generalized Smoothness

Beyond smoothness: A general optimization framework for graph neural networks with negative Laplacian regularization

Review Non-convex Optimization Method for Machine Learning

Smoothing Neural Network for Constrained Non-Lipschitz Optimization with Applications

Adaptive Methods for Variational Inequalities under Relaxed Smoothness Assumption

Non-convex Optimization for Machine Learning

A gentle introduction to gradient-based optimization and variational inequalities for machine learning

Modeling the Nonsmoothness of Modern Neural Networks

Analysis of Gradient Clipping and Adaptive Scaling with a Relaxed Smoothness Condition

Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity

Non-Convex Optimizations for Machine Learning with Theoretical Guarantee: Robust Matrix Completion and Neural Network Learning

Gradient-Variation Online Learning under Generalized Smoothness

The Challenges of Optimization For Data Science

Neural Network for Nonsmooth Pseudoconvex Optimization with General Convex Constraints

Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization

Neural Network for Nonsmooth, Nonconvex Constrained Minimization Via Smooth Approximation.

Subgradient-based Neural Networks for Nonsmooth Nonconvex Optimization Problems

An Analysis of a Neural Dynamical Approach to Solving Optimization Problems

The role of optimization geometry in single neuron learning