Abstract:Deep learning approaches have provided state-of-the-art performance in many applications by relying on large and overparameterized neural networks. However, such networks have been shown to be very brittle and are difficult to deploy on resource-limited platforms. Model pruning, i.e., reducing the size of the network, is a widely adopted strategy that can lead to a more robust and compact model. Many heuristics exist for model pruning, but empirical studies show that some heuristics improve performance whereas others can make models more brittle or have other side effects. This work aims to shed light on how different pruning methods alter the network's internal feature representation and the corresponding impact on model performance. To facilitate a comprehensive comparison and characterization of the high-dimensional model feature space, we introduce a visual geometric analysis of feature representations. We decomposed and evaluated a set of critical geometric concepts from the common adopted classification loss, and used them to design a visualization system to compare and highlight the impact of pruning on model performance and feature representation. The proposed tool provides an environment for in-depth comparison of pruning methods and a comprehensive understanding of how model response to common data corruption. By leveraging the proposed visualization, machine learning researchers can reveal the similarities between pruning methods and redundant in robustness evaluation benchmarks, obtain geometric insights about the differences between pruned models that achieve superior robustness performance, and identify samples that are robust or fragile to model pruning and common data corruption to model pruning and data corruption but also obtain insights and explanations on how some pruned models achieve superior robustness performance.

Information Geometry on Pruning of Neural Network

Class-Aware Pruning for Efficient Neural Networks

A Feature-map Discriminant Perspective for Pruning Deep Neural Networks

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Causal Unstructured Pruning in Linear Networks Using Effective Information.

Statistical Mechanical Analysis of Neural Network Pruning

Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importance

Manifold Regularized Dynamic Network Pruning

Neural Network Pruning by Gradient Descent

Structural Pruning in Deep Neural Networks: A Small-World Approach

Network Pruning Spaces

Complexity-Aware Training of Deep Neural Networks for Optimal Structure Discovery

Network Pruning Via Transformable Architecture Search.

Joint Multi-Dimension Pruning via Numerical Gradient Update

LSOP: Layer-Scaled One-shot Pruning

Fine Granularity Is Critical for Intelligent Neural Network Pruning

A Probabilistic Approach to Neural Network Pruning

Investigating the Effect of Network Pruning on Performance and Interpretability

Mutual Information Preserving Neural Network Pruning

"Understanding Robustness Lottery": A Geometric Visual Comparative Analysis of Neural Network Pruning Approaches

Knapsack Pruning with Inner Distillation