Abstract:In this work, we consider convex optimization problems with smooth objective function and nonsmooth functional constraints. We propose a new stochastic gradient algorithm, called Stochastic Halfspace Approximation Method (SHAM), to solve this problem, where at each iteration we first take a gradient step for the objective function and then we perform a projection step onto one halfspace approximation of a randomly chosen constraint. We propose various strategies to create this stochastic halfspace approximation and we provide a unified convergence analysis that yields new convergence rates for SHAM algorithm in both optimality and feasibility criteria evaluated at some average point. In particular, we derive convergence rates of order $\mathcal{O} (1/\sqrt{k})$, when the objective function is only convex, and $\mathcal{O} (1/k)$ when the objective function is strongly convex. The efficiency of SHAM is illustrated through detailed numerical simulations.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the convex optimization problem with non - smooth function constraints. Specifically, the objective function of this type of problem is smooth, but the constraint conditions are non - smooth. Traditional methods face challenges when dealing with a large number of constraint conditions or when projection calculations are complex. Therefore, the author proposes a new stochastic gradient algorithm - the Stochastic Halfspace Approximation Method (SHAM) to solve this type of optimization problem more efficiently. ### Problem Description Consider the following convex optimization problem: \[ f^*=\min_{x\in Y\subseteq\mathbb{R}^n}f(x) \] Subject to: \[ h_j(x)\leq0,\quad\forall j\in[m] \] Where: - $f:\mathbb{R}^n\rightarrow\mathbb{R}$ is the objective function and is $L_f$-smooth; - $h_j:\mathbb{R}^n\rightarrow\mathbb{R}$ are non - smooth convex function constraints; - $Y$ is a non - empty, closed, convex and simple set (i.e., easy to project). When the number of constraints $m$ is large, or when the projection of certain subsets is difficult to calculate, these problems can become very computationally intensive or even impossible to solve. Therefore, a new method is proposed to meet these challenges. ### Solution The SHAM algorithm solves the above problem through the following steps: 1. **Gradient Step**: Perform a gradient step on the objective function. 2. **Projection Step**: Project the intermediate point onto the linear half - space approximation of a randomly selected constraint. The main contributions of SHAM include: - Proposing multiple methods for creating linear half - space approximations and providing a unified convergence analysis. - For convex objective functions, deriving a sub - linear convergence rate of $O(1/\sqrt{k})$, and for strongly convex objective functions, deriving a convergence rate of $O(1/k)$. - Proposing a switching step - size strategy to transition from a constant step - size to a non - increasing step - size to adapt to different optimization requirements. The efficiency and practicality of SHAM in practical applications are verified through numerical experiments.

Stochastic halfspace approximation method for convex optimization with nonsmooth functional constraints

A stochastic moving ball approximation method for smooth convex constrained minimization

Stochastic Successive Convex Approximation for Non-Convex Constrained Stochastic Optimization

Stochastic subgradient for composite optimization with functional constraints

Stochastic proximal gradient methods for nonconvex problems in Hilbert spaces

Solving a Class of Convex Stochastic Programs Via Monte Carlo Simulation

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

Stochastic smoothing accelerated gradient method for nonsmooth convex composite optimization

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

Stochastic Approximation Proximal Subgradient Method for Stochastic Convex-Concave Minimax Optimization

Stochastic Optimization for Non-convex Problem with Inexact Hessian Matrix, Gradient, and Function

Two Stochastic Optimization Algorithms for Convex Optimization with Fixed Point Constraints

Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

Stochastic Approximation in convex multiobjective optimization

Stochastic alternating structure-adapted proximal gradient descent method with variance reduction for nonconvex nonsmooth optimization

A Descent Method for Nonsmooth Multiobjective Optimization in Hilbert Spaces

A stochastic regularized second-order iterative scheme for optimal control and inverse problems in stochastic partial differential equations

Beyond Convexity: Stochastic Quasi-Convex Optimization