Abstract:Layer pruning offers a promising alternative to standard structured pruning, effectively reducing computational costs, latency, and memory footprint. While notable layer-pruning approaches aim to detect unimportant layers for removal, they often rely on single criteria that may not fully capture the complex, underlying properties of layers. We propose a novel approach that combines multiple similarity metrics into a single expressive measure of low-importance layers, called the Consensus criterion. Our technique delivers a triple-win solution: low accuracy drop, high-performance improvement, and increased robustness to adversarial attacks. With up to 78.80% FLOPs reduction and performance on par with state-of-the-art methods across different benchmarks, our approach reduces energy consumption and carbon emissions by up to 66.99% and 68.75%, respectively. Additionally, it avoids shortcut learning and improves robustness by up to 4 percentage points under various adversarial attacks. Overall, the Consensus criterion demonstrates its effectiveness in creating robust, efficient, and environmentally friendly pruned models.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to maintain the high prediction ability and robustness of deep - learning models while reducing computational cost, latency and memory usage. Specifically, the authors focus on the layer - pruning technique, aiming to achieve this goal by removing unimportant network layers. ### Main problems: 1. **High computational cost and memory consumption**: Although over - parameterized deep - learning models can represent complex data patterns, their high computational cost and memory consumption limit their applications in low - resource environments. 2. **Vulnerability to adversarial attacks**: High - capacity models are vulnerable to adversarial samples, leading to prediction errors, which makes them unreliable in tasks with high security and reliability requirements. 3. **Limitations of existing pruning methods**: Existing layer - pruning methods usually rely on a single criterion to identify and remove unimportant layers, and these criteria may not fully capture the complex characteristics of layers, thus affecting the accuracy and robustness of the model. ### The method proposed in the paper: To solve the above problems, the authors propose a new layer - pruning method - the **Consensus criterion**. This method combines multiple similarity measures to form a comprehensive low - importance - layer evaluation criterion. Compared with methods that rely only on a single criterion, the Consensus criterion has the following advantages: - **Low accuracy drop**: By integrating multiple similarity measures, this method can maintain high model accuracy at a high compression rate. - **High performance improvement**: Effectively prune unimportant layers, significantly reduce computational cost, and improve inference time and memory - use efficiency. - **Enhanced robustness against adversarial attacks**: Avoid the bias brought by relying on a single measure, make the model more robust, and better resist various adversarial perturbations. ### Experimental results: Experiments show that this method has achieved excellent performance on multiple benchmark datasets such as CIFAR - 10 and ImageNet. In particular, when the FLOPs are reduced by up to 78.80%, the model still maintains a low accuracy drop, and its performance under adversarial attacks is better than that of existing methods. ### Summary: The Consensus criterion proposed in the paper provides a three - win solution: low accuracy drop, high performance improvement, and enhanced robustness against adversarial attacks. This makes the pruned model not only more efficient, but also more reliable and environmentally friendly.

Layer Pruning with Consensus: A Triple-Win Solution

Class-Aware Pruning for Efficient Neural Networks

Reassessing Layer Pruning in LLMs: New Insights and Methods

Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks

To Filter Prune, or to Layer Prune, That Is the Question

Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks

The Unreasonable Ineffectiveness of the Deeper Layers

Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint

LSOP: Layer-Scaled One-shot Pruning

When Layers Play the Lottery, all Tickets Win at Initialization

Layer-adaptive Structured Pruning Guided by Latency

Learning a Consensus Sub-Network with Polarization Regularization and One Pass Training

FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models

Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity

Concurrent Training and Layer Pruning of Deep Neural Networks

Consensus Community Detection in Multilayer Networks using Parameter-free Graph Pruning

Pruning Foundation Models for High Accuracy without Retraining

Structured Pruning Learns Compact and Accurate Models

Layer-adaptive sparsity for the Magnitude-based Pruning

Cross-layer importance evaluation for neural network pruning

LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging