Abstract:This paper proposes a novel technique called "successive stochastic smoothing" that optimizes nonsmooth and discontinuous functions while considering various constraints. Our methodology enables local and global optimization, making it a powerful tool for many applications. First, a constrained problem is reduced to an unconstrained one by the exact nonsmooth penalty function method, which does not assume the existence of the objective function outside the feasible area and does not require the selection of the penalty coefficient. This reduction is exact in the case of minimization of a lower semicontinuous function under convex constraints. Then the resulting objective function is sequentially smoothed by the kernel method starting from relatively strong smoothing and with a gradually vanishing degree of smoothing. The finite difference stochastic gradient descent with trajectory averaging minimizes each smoothed function locally. Finite differences over stochastic directions sampled from the kernel estimate the stochastic gradients of the smoothed functions. We investigate the convergence rate of such stochastic finite-difference method on convex optimization problems. The "successive smoothing" algorithm uses the results of previous optimization runs to select the starting point for optimizing a consecutive, less smoothed function. Smoothing provides the "successive smoothing" method with some global properties. We illustrate the performance of the "successive stochastic smoothing" method on test-constrained optimization problems from the literature.

Smoothing Neural Network for Constrained Non-Lipschitz Optimization with Applications

An Improved Smoothing Technique-Based Control Vector Parameterization Method for Optimal Control Problems with Inequality Path Constraints

A discrete-time neural network for optimization problems with hybrid constraints.

A New Neural Network Model And Its Application

Adaptive Smoothing Gradient Learning for Spiking Neural Networks.

A neural network model for non-smooth optimization over a compact convex subset

Constrained Global Optimization by Smoothing

Beyond smoothness: A general optimization framework for graph neural networks with negative Laplacian regularization

Differential inclusions-based neural networks for nonsmooth convex optimization on a closed convex subset

Generalized Gradient Projection Neural Networks for Nonsmooth Optimization Problems

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

A neural network for constrained saddle point problems: an approximation approach

A fast algorithm for solving large scale nonlinear optimization problems using RNN

Locally Smoothed Neural Networks.

A variable smoothing for Nonconvexly constrained nonsmooth optimization with application to sparse spectral clustering

A Neural Network Based on a Nonsmooth Equation for a Box Constrained Variational Inequality Problem

Mini-batch stochastic Nesterov's smoothing method for constrained convex stochastic composite optimization

Enhancing neurodynamic approach with physics-informed neural networks for solving non-smooth convex optimization problems

Image smoothing via unsupervised learning

Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees

Recent Advances in Non-convex Smoothness Conditions and Applicability to Deep Linear Neural Networks