Abstract:This paper leverages the use of \emph{Gram iteration} an efficient, deterministic, and differentiable method for computing spectral norm with an upper bound guarantee. Designed for circular convolutional layers, we generalize the use of the Gram iteration to zero padding convolutional layers and prove its quadratic convergence. We also provide theorems for bridging the gap between circular and zero padding convolution's spectral norm. We design a \emph{spectral rescaling} that can be used as a competitive $1$-Lipschitz layer that enhances network robustness. Demonstrated through experiments, our method outperforms state-of-the-art techniques in precision, computational cost, and scalability. The code of experiments is available at
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: **How to calculate the spectral norm of the convolutional layer in the convolutional neural network (CNN) efficiently and accurately, and provide a strict upper - bound estimate to enhance the robustness and generalization ability of the network**.
Specifically, the paper focuses on the following aspects:
1. **Importance of spectral norm**:
- The spectral norm plays a crucial role in understanding the generalization ability and robustness of neural networks. Especially for the convolutional layer, the boundary of the spectral norm can improve the generalization performance of the model and enhance the robustness against adversarial attacks.
- The paper points out that existing methods such as power iteration and singular value decomposition (SVD) can estimate the spectral norm, but lack strict upper - bound guarantees, which are crucial for robustness analysis and certification.
2. **Limitations of existing methods**:
- The spectral norm calculation methods (such as SVD) directly applied to the convolutional layer are computationally expensive and difficult to use during the training process.
- Existing methods such as power iteration can estimate the spectral norm, but cannot provide a strict upper - bound, resulting in the inability to reliably calculate the worst - case changes, thus affecting the accuracy of the robustness certificate.
3. **Application and improvement of Gram iteration**:
- The author introduces the Gram iteration method, which is an efficient, deterministic, and differentiable method that can provide a strict upper - bound estimate for the spectral norm.
- By extending the Gram iteration method, the author not only deals with the circular padding convolutional layer but also generalizes it to the zero - padding convolutional layer and proves its quadratic convergence.
4. **Theoretical and experimental verification**:
- The paper provides detailed theoretical proofs, including the quadratic convergence of Gram iteration and the approximation method of the upper - bound of the spectral norm of the convolutional layer for different input sizes.
- The experimental results show that the proposed method is superior to the existing techniques in terms of accuracy, computational cost, and scalability, especially when dealing with large - scale images.
5. **Application and innovation**:
- The author designs a new 1 - Lipschitz layer, using the Spectral Rescaling (SR) method, making the network more robust.
- The SR method can not only converge to a spectral norm of 1 but also maintain a good condition number, avoiding the ill - conditioned matrix problem brought by spectral normalization (SN).
In summary, this paper aims to provide an efficient and strict upper - bound estimation method for the spectral norm calculation of the convolutional layer by improving the Gram iteration method, thereby enhancing the robustness and generalization ability of the convolutional neural network.