Abstract:This work analyzes the convergence of a class of smoothing-based gradient descent methods when applied to optimization problems. In particular, Gaussian smoothing is employed to define a nonlocal gradient that reduces high-frequency noise, small variations, and rapid fluctuations in the computation of the descent directions while preserving the structure and features of the loss landscape. The resulting Gaussian smoothing gradient descent (GSmoothGD) approach can facilitate gradient descent in navigating away from and avoiding local minima with increased ease, thereby substantially enhancing its overall performance even when applied to non-convex optimization problems. This work also provides rigorous theoretical error estimates on the rate of convergence of GSmoothGD iterates. These estimates exemplify the impact of underlying function convexity, smoothness, input dimension, and the Gaussian smoothing radius. To combat the curse of dimensionality, we numerically approximate the GSmoothGD nonlocal gradient using Monte Carlo (MC) sampling and provide a theory in which the iterates converge regardless of the function smoothness and dimension. Finally, we present several strategies to update the smoothing parameter aimed at diminishing the impact of local minima, thereby rendering the attainment of global minima more achievable. Computational evidence complements the present theory and shows the effectiveness of the MC-GSmoothGD method compared to other smoothing-based algorithms, momentum-based approaches, and classical gradient-based algorithms from numerical optimization.

Constrained Global Optimization by Smoothing

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Stochastic smoothing accelerated gradient method for nonsmooth convex composite optimization

A stochastic moving ball approximation method for smooth convex constrained minimization

A Smoothing Stochastic Gradient Method for Composite Optimization

A Variable Smoothing for Weakly Convex Composite Minimization with Nonconvex Constraint

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

A feasible smoothing accelerated projected gradient method for nonsmooth convex optimization

A Smoothing Direct Search Method for Monte Carlo-Based Bound Constrained Composite Nonsmooth Optimization.

Smoothing Neural Network for Constrained Non-Lipschitz Optimization with Applications

Smoothing algorithms for nonsmooth and nonconvex minimization over the stiefel manifold

Convex and Non-convex Optimization Under Generalized Smoothness

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

Gaussian smoothing gradient descent for minimizing functions (GSmoothGD)

Mini-batch stochastic Nesterov's smoothing method for constrained convex stochastic composite optimization

Smoothing Algorithms for Nonsmooth and Nonconvex Minimization over the Stiefel Manifold

Distributed Smoothing Projection Neurodynamic Approaches for Constrained Nonsmooth Optimization

A Penalty-Free Infeasible Approach for a Class of Nonsmooth Optimization Problems Over the Stiefel Manifold