Abstract:Recent results show that vanilla gradient descent can be accelerated for smooth convex objectives, merely by changing the stepsize sequence. We show that this can lead to surprisingly large errors indefinitely, and therefore ask: Is there any stepsize schedule for gradient descent that accelerates the classic $\mathcal{O}(1/T)$ convergence rate, at \emph{any} stopping time $T$?
What problem does this paper attempt to address?
This paper explores the convergence rate at any time of the Gradient Descent (GD) method when optimizing smooth convex objective functions. Specifically, the paper attempts to solve the following problems:
### Problems the Paper Attempts to Solve
**Problem 1: Is there a certain step - length sequence such that the Gradient Descent method can accelerate the classical \(O(1/T)\) convergence rate at any stopping time \(T\)?**
Specifically, the paper proposes an open problem (Open Problem):
- **Open Problem 1**: For any \(L\)-smooth convex function \(f\), is there a step - length sequence \((\eta_t)_{t = 0}^{\infty}\) such that the Gradient Descent method satisfies the following condition at any stopping time \(T\):
\[
f(x_T)-f^*\lesssim\frac{L\|x_0 - x^*\|^2}{T^{\alpha}}\quad\text{for all}\;T\in\mathbb{N},
\]
where \(\alpha> 1\).
### Background and Motivation
1. **Convergence Rate of the Traditional Gradient Descent Method**:
- Classical analysis shows that when the step - length is fixed as \(\eta_t\equiv\eta\in(0,2/L)\), the Gradient Descent method satisfies the following after \(T\) iterations:
\[
f(x_T)-f^*\lesssim\frac{L\|x_0 - x^*\|^2}{T}.
\]
2. **Recent Research Results**:
- Recent research has found that by using an appropriate non - constant step - length sequence, the convergence rate of the Gradient Descent method can be accelerated. For example, the "silver stepsize" sequence proposed by Altschuler and Parrilo can achieve a convergence rate of \(O(1/T^{1.2716})\) at certain specific time points \(T = 2^n-1\).
3. **Problems in Practical Applications**:
- However, these acceleration methods are only effective at specific time points and are not guaranteed at other time points. From the perspective of practical applications, this is not ideal because the number of iterations is usually not precisely determined in advance. Therefore, researchers hope to find a method that can maintain accelerated convergence at any stopping time \(T\).
### Main Contributions
1. **Theoretical Analysis**:
- Through two preliminary results, the author shows the challenges in accelerating the Gradient Descent method at any time. In particular, they prove that:
- If the Gradient Descent method can accelerate convergence at any time \(T\), the step - length sequence must contain arbitrarily large step - lengths (Theorem 1).
- Occasional large step - lengths may lead to a significant increase in error, thereby destroying any consistent convergence guarantee (Theorem 2).
2. **Conclusion**:
- The author points out that the existing accelerated step - length sequences (such as "silver stepsize") cannot provide a consistent convergence guarantee at any time. Therefore, they propose the above open problem, hoping to stimulate more research on this topic.
### Summary
This paper aims to explore the problem of accelerated convergence of the Gradient Descent method at any stopping time, and proposes a series of theoretical results and open problems to promote further research in this field.