Layer Pruning with Consensus: A Triple-Win Solution

Leandro Giusti Mugnaini,Carolina Tavares Duarte,Anna H. Reali Costa,Artur Jordao
2024-11-22
Abstract:Layer pruning offers a promising alternative to standard structured pruning, effectively reducing computational costs, latency, and memory footprint. While notable layer-pruning approaches aim to detect unimportant layers for removal, they often rely on single criteria that may not fully capture the complex, underlying properties of layers. We propose a novel approach that combines multiple similarity metrics into a single expressive measure of low-importance layers, called the Consensus criterion. Our technique delivers a triple-win solution: low accuracy drop, high-performance improvement, and increased robustness to adversarial attacks. With up to 78.80% FLOPs reduction and performance on par with state-of-the-art methods across different benchmarks, our approach reduces energy consumption and carbon emissions by up to 66.99% and 68.75%, respectively. Additionally, it avoids shortcut learning and improves robustness by up to 4 percentage points under various adversarial attacks. Overall, the Consensus criterion demonstrates its effectiveness in creating robust, efficient, and environmentally friendly pruned models.
Machine Learning,Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is how to maintain the high prediction ability and robustness of deep - learning models while reducing computational cost, latency and memory usage. Specifically, the authors focus on the layer - pruning technique, aiming to achieve this goal by removing unimportant network layers. ### Main problems: 1. **High computational cost and memory consumption**: Although over - parameterized deep - learning models can represent complex data patterns, their high computational cost and memory consumption limit their applications in low - resource environments. 2. **Vulnerability to adversarial attacks**: High - capacity models are vulnerable to adversarial samples, leading to prediction errors, which makes them unreliable in tasks with high security and reliability requirements. 3. **Limitations of existing pruning methods**: Existing layer - pruning methods usually rely on a single criterion to identify and remove unimportant layers, and these criteria may not fully capture the complex characteristics of layers, thus affecting the accuracy and robustness of the model. ### The method proposed in the paper: To solve the above problems, the authors propose a new layer - pruning method - the **Consensus criterion**. This method combines multiple similarity measures to form a comprehensive low - importance - layer evaluation criterion. Compared with methods that rely only on a single criterion, the Consensus criterion has the following advantages: - **Low accuracy drop**: By integrating multiple similarity measures, this method can maintain high model accuracy at a high compression rate. - **High performance improvement**: Effectively prune unimportant layers, significantly reduce computational cost, and improve inference time and memory - use efficiency. - **Enhanced robustness against adversarial attacks**: Avoid the bias brought by relying on a single measure, make the model more robust, and better resist various adversarial perturbations. ### Experimental results: Experiments show that this method has achieved excellent performance on multiple benchmark datasets such as CIFAR - 10 and ImageNet. In particular, when the FLOPs are reduced by up to 78.80%, the model still maintains a low accuracy drop, and its performance under adversarial attacks is better than that of existing methods. ### Summary: The Consensus criterion proposed in the paper provides a three - win solution: low accuracy drop, high performance improvement, and enhanced robustness against adversarial attacks. This makes the pruned model not only more efficient, but also more reliable and environmentally friendly.