Abstract:We present TropNNC, a structured pruning framework for compressing neural networks with linear and convolutional layers and ReLU activations. Our approximation is based on a geometrical approach to machine/deep learning, using tropical geometry and extending the work of Misiakos et al. (2022). We use the Hausdorff distance of zonotopes in its standard continuous form to achieve a tighter approximation bound for tropical polynomials compared to Misiakos et al. (2022). This enhancement allows for superior functional approximations of neural networks, leading to a more effective compression algorithm. Our method is significantly easier to implement compared to other frameworks, and does not depend on the availability of training data samples. We validate our framework through extensive empirical evaluations on the MNIST, CIFAR, and ImageNet datasets. Our results demonstrate that TropNNC achieves performance on par with the state-of-the-art method ThiNet, even surpassing it in compressing linear layers, and to the best of our knowledge, it is the first method that achieves this using tropical geometry.

What problem does this paper attempt to address?

This paper attempts to solve the problem of excessive computational and storage requirements when neural networks are deployed on resource - constrained devices. Specifically, the authors propose TropNNC, a structured neural network compression framework based on tropical geometry. This framework aims to effectively compress neural networks with linear and convolutional layers as well as ReLU activation functions. ### Main Problems and Solutions 1. **Problem Background**: - Deep neural networks (DNNs) perform excellently in computer vision tasks, but their huge computational and storage requirements make them difficult to be deployed in resource - constrained environments such as mobile devices or embedded systems. - For example, the VGG - 16 model contains approximately 138 million parameters, requires more than 500MB of storage space, and classifying a single image requires approximately 31 billion floating - point operations (FLOPs). Such resource - intensive models exceed the capabilities of small devices and pose significant challenges, such as energy limitations. 2. **Limitations of Existing Methods**: - **Unstructured Pruning**: Reducing the network scale by removing individual weights is effective, but in practical applications, it faces problems such as poor cache locality and jumpy memory access, which may lead to performance degradation. - **Structured Pruning**: For example, the ThiNet framework maintains the original structure of the network by pruning entire filters or channels, thereby ensuring compatibility with existing deep - learning libraries and providing better compression and acceleration effects. 3. **Innovations of TropNNC**: - **Using Tropical Geometry**: Utilizing zonotope and Hausdorff distance in tropical geometry to achieve a closer functional approximation, thereby improving the effectiveness of the compression algorithm. - **Improved Error Bounds**: Obtaining tighter error bounds than Misiakos et al. (2022) through the standard continuous form of Hausdorff distance. - **No Need for Training Data**: This method does not rely on training data samples, simplifying the implementation process. - **First Application to Convolutional Layers**: This is the first method to compress convolutional layers based on tropical geometry theory. ### Experimental Verification The authors prove the effectiveness of TropNNC through extensive experiments on datasets such as MNIST, CIFAR - 10, CIFAR - 100, and ImageNet. The experimental results show that TropNNC outperforms the existing state - of - the - art method ThiNet in compressing linear layers, and its performance on the VGG architecture is particularly prominent. ### Summary TropNNC provides a new perspective and tool for neural network compression by introducing tropical geometry, which not only improves the compression efficiency but also maintains high precision. This provides new possibilities for the efficient deployment of deep neural networks on resource - constrained devices.

TropNNC: Structured Neural Network Compression Using Tropical Geometry

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression

Neural Network Compression by Joint Sparsity Promotion and Redundancy Reduction

Advances in the training, pruning and enforcement of shape constraints of Morphological Neural Networks using Tropical Algebra

Enhanced Network Compression Through Tensor Decompositions and Pruning

Pruning at a Glance: Global Neural Pruning for Model Compression

Towards Compact CNNs via Collaborative Compression

Efficient Neural Network Compression Inspired by Compressive Sensing.

Neural Network Compression Framework for Fast Model Inference

Tropical Expressivity of Neural Networks

Nonlinear Tensor Train Format for Deep Neural Network Compression

Hybrid Tensor Decomposition in Neural Network Compression

An Alternative Practice of Tropical Convolution to Traditional Convolutional Neural Networks

Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition

ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression.

ECNR: Efficient Compressive Neural Representation of Time-Varying Volumetric Datasets

End-to-End Neural Network Compression via $\frac{\ell_1}{\ell_2}$ Regularized Latency Surrogates

On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee

3D-KCPNet: Efficient 3DCNNs based on tensor mapping theory

Neural Network Compression Via Sparse Optimization