Abstract:This paper proposes a novel technique called "successive stochastic smoothing" that optimizes nonsmooth and discontinuous functions while considering various constraints. Our methodology enables local and global optimization, making it a powerful tool for many applications. First, a constrained problem is reduced to an unconstrained one by the exact nonsmooth penalty function method, which does not assume the existence of the objective function outside the feasible area and does not require the selection of the penalty coefficient. This reduction is exact in the case of minimization of a lower semicontinuous function under convex constraints. Then the resulting objective function is sequentially smoothed by the kernel method starting from relatively strong smoothing and with a gradually vanishing degree of smoothing. The finite difference stochastic gradient descent with trajectory averaging minimizes each smoothed function locally. Finite differences over stochastic directions sampled from the kernel estimate the stochastic gradients of the smoothed functions. We investigate the convergence rate of such stochastic finite-difference method on convex optimization problems. The "successive smoothing" algorithm uses the results of previous optimization runs to select the starting point for optimizing a consecutive, less smoothed function. Smoothing provides the "successive smoothing" method with some global properties. We illustrate the performance of the "successive stochastic smoothing" method on test-constrained optimization problems from the literature.

A feasible smoothing accelerated projected gradient method for nonsmooth convex optimization

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

Stochastic smoothing accelerated gradient method for nonsmooth convex composite optimization

Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization

Accelerated forward-backward method with fast convergence rate for nonsmooth convex optimization beyond differentiability

An Accelerated Distributed Method with Inexact Model of Relative Smoothness and Strong Convexity

A Nesterov's Accelerated Projected Gradient Method for Monotone Variational Inequalities

An Infeasible-Point Subgradient Method Using Adaptive Approximate Projections

O(log T) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions

Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

Smoothing Accelerated Proximal Gradient Method with Fast Convergence Rate for Nonsmooth Multi-objective Optimization

Smoothing Accelerated Proximal Gradient Method with Fast Convergence Rate for Nonsmooth Convex Optimization Beyond Differentiability

Accelerated Projection Algorithm Based on Smoothing Approximation for Distributed Nonsmooth Optimization

Accelerated Projection Algorithm Based on Smoothing Approximation for Distributed Non-Smooth Optimization

Accelerated gradient methods for nonconvex nonlinear and stochastic programming

A Smoothing Stochastic Gradient Method for Composite Optimization

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

Nonsmooth Projection-Free Optimization with Functional Constraints

Constrained Global Optimization by Smoothing

An Average Curvature Accelerated Composite Gradient Method for Nonconvex Smooth Composite Optimization Problems

Linear convergence of forward-backward accelerated algorithms without knowledge of the modulus of strong convexity