Robust low-rank training via approximate orthonormal constraints

Dayana Savostianova,Emanuele Zangrando,Gianluca Ceruti,Francesco Tudisco

2023-06-02

Abstract:With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to represent the network weights. Although able to retain accuracy, we observe that low-rank methods tend to compromise model robustness against adversarial perturbations. By modeling robustness in terms of the condition number of the neural network, we argue that this loss of robustness is due to the exploding singular values of the low-rank weight matrices. Thus, we introduce a robust low-rank training algorithm that maintains the network's weights on the low-rank matrix manifold while simultaneously enforcing approximate orthonormal constraints. The resulting model reduces both training and inference costs while ensuring well-conditioning and thus better adversarial robustness, without compromising model accuracy. This is shown by extensive numerical evidence and by our main approximation theorem that shows the computed robust low-rank network well-approximates the ideal full model, provided a highly performing low-rank sub-network exists.

Machine Learning,Artificial Intelligence,Numerical Analysis

What problem does this paper attempt to address?

The paper aims to address the issue of maintaining model performance in deep learning while reducing resource requirements, particularly robustness under adversarial perturbations. Specifically, the paper focuses on the following points:

1. **Reducing Training and Inference Costs**: Representing network weights through low-rank matrix decomposition to reduce the costs of training and inference. Although this method can retain model accuracy, it often sacrifices the model's robustness to adversarial perturbations.

2. **Improving Adversarial Robustness**: The paper points out that low-rank methods lead to an increase in the model's condition number, thereby reducing its adversarial robustness. To address this, the authors propose a new robust low-rank training algorithm that enforces approximate orthogonal constraints while maintaining the low-rank structure of the weights, ensuring a good condition number for the model and improving adversarial robustness.

3. **Theoretical Support**: Through extensive numerical experiments and a key approximation theorem, it is demonstrated that the proposed robust low-rank network can significantly reduce the memory and computational costs required for training and inference without sacrificing model accuracy, and maintain or improve the robust accuracy of the original model.

In summary, the main goal of the paper is to ensure the robustness of the model, especially under adversarial attacks, while reducing the costs of model training and inference.