Abstract:This paper considers stochastic weakly convex optimization without the standard Lipschitz continuity assumption. Based on new adaptive regularization (stepsize) strategies, we show that a wide class of stochastic algorithms, including the stochastic subgradient method, preserve the $\mathcal{O} ( 1 / \sqrt{K})$ convergence rate with constant failure rate. Our analyses rest on rather weak assumptions: the Lipschitz parameter can be either bounded by a general growth function of $\|x\|$ or locally estimated through independent random samples.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to conduct Stochastic Weakly Convex Optimization without the standard Lipschitz continuity assumption. Specifically, the author focuses on how to design effective algorithms to maintain good convergence performance when the function may not satisfy the global Lipschitz condition. Traditionally, many optimization algorithms rely on the Lipschitz continuity assumption to guarantee their convergence, but in some practical problems, this assumption is too strict, causing the algorithms to be potentially unstable or even divergent. Therefore, this paper proposes a new adaptive step - size strategy, aiming to overcome this limitation, so that even under non - Lipschitz conditions, an \(O(1/\sqrt{K})\) convergence rate can be achieved with a constant failure probability.
### Main contributions of the paper
1. **Convergence under any non - Lipschitz objective**:
- When the Lipschitz constant is not uniformly bounded but depends on a general growth function \(G(\cdot)\), the author designs a new adaptive step - size strategy, enabling Stochastic Weakly Convex Optimization to still achieve an \(O(1/\sqrt{K})\) convergence rate under a constant failure probability. This result is applicable to a wide range of model - based algorithms, including SGD as a special case.
2. **Convergence under an unknown growth function**:
- Even when the growth function \(G\) is unknown, the author also shows how to estimate the local Lipschitz parameter by introducing the concept of "reference Lipschitz continuity", thereby achieving the same convergence guarantee. This adaptive step - size strategy is very flexible and can be applied to most weakly convex problems of interest.
### Main technical means
- **Adaptive step - size strategy**: Dynamically adjust the step - size according to the properties of the current iteration point to cope with the challenges brought by non - Lipschitzness.
- **Reference Lipschitz continuity**: Estimate the Lipschitz constant through independent samples to avoid the bias caused by directly using related samples.
- **Probability analysis**: Use probability tools to analyze the stability of the iteration point and ensure that the algorithm converges with high probability.
### Experimental verification
The author verifies the effectiveness and robustness of the proposed method through numerical experiments. The experiments include different types of nonlinear regression problems, which show different degrees of non - Lipschitzness. The experimental results show that for low - order growth functions, the adaptive step - size strategy may be relatively conservative; while for high - order growth functions, the adaptive step - size strategy can converge within a reasonable step - size range. In addition, the adaptive step - size strategy based on reference Lipschitz continuity never diverges in practice, although it sometimes appears conservative in problems with milder function growth.
### Conclusion
This paper successfully solves the problem of conducting Stochastic Weakly Convex Optimization under non - Lipschitz conditions by proposing a new adaptive step - size strategy. This result not only expands the scope of application of weakly convex optimization but also provides new ideas and methods for dealing with complex optimization problems in practice.