Electrostatic Force Regularization for Neural Structured Pruning

Abdesselam Ferdi,Abdelmalik Taleb-Ahmed,Amir Nakib,Youcef Ferdi

2024-11-17

Abstract:The demand for deploying deep convolutional neural networks (DCNNs) on resource-constrained devices for real-time applications remains substantial. However, existing state-of-the-art structured pruning methods often involve intricate implementations, require modifications to the original network architectures, and necessitate an extensive fine-tuning phase. To overcome these challenges, we propose a novel method that, for the first time, incorporates the concepts of charge and electrostatic force from physics into the training process of DCNNs. The magnitude of this force is directly proportional to the product of the charges of the convolution filter and the source filter, and inversely proportional to the square of the distance between them. We applied this electrostatic-like force to the convolution filters, either attracting filters with opposite charges toward non-zero weights or repelling filters with like charges toward zero weights. Consequently, filters subject to repulsive forces have their weights reduced to zero, enabling their removal, while the attractive forces preserve filters with significant weights that retain information. Unlike conventional methods, our approach is straightforward to implement, does not require any architectural modifications, and simultaneously optimizes weights and ranks filter importance, all without the need for extensive fine-tuning. We validated the efficacy of our method on modern DCNN architectures using the MNIST, CIFAR, and ImageNet datasets, achieving competitive performance compared to existing structured pruning approaches.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the challenges faced when deploying deep convolutional neural networks (DCNNs) on resource - constrained devices for real - time applications. Specifically, the existing structured pruning methods usually involve complex implementation, require modification of the original network architecture, and need a long - term fine - tuning phase. These problems make it difficult for the existing methods to be efficiently applied in practical scenarios.

To overcome these challenges, the author proposes a novel method, which for the first time introduces the concepts of charge and electrostatic force in physics into the training process of DCNNs. Through this method, the author hopes to achieve the following goals:

1. **Simplify implementation**: The new method does not require any modification of the network architecture, thus simplifying the implementation process.
2. **Optimize the importance of weights and filters**: Optimize the weights and determine the importance of filters simultaneously without extensive fine - tuning.
3. **Reduce model complexity**: By pruning unimportant filters, reduce model parameters and the number of floating - point operations (FLOPs), thereby reducing memory requirements and energy consumption while minimizing the loss of accuracy.

### Method overview

The main idea of the electrostatic - force - based structured pruning method (Electrostatic Force Regularization for Neural Structured Pruning, EFR - NSP) proposed by the author is to use electrostatic force to adjust the weights of filters in the convolutional layer. Specifically:

- **Electrostatic force formula**:
According to Coulomb's law, the magnitude of the electrostatic force \( F_e \) between two charges is:
\[
F_e = k_e\frac{|q_1||q_n|}{r^2}
\]
where \( k_e \) is the Coulomb constant (\( 8.99\times 10^9 \, \text{Nm}^2\text{C}^{-2} \)), \( q_1 \) and \( q_n \) are the charges of the source filter and the \( n \) - th filter respectively, and \( r \) is the distance between them.

- **Charge definition**:
The charge \( q_{n,l} \) of a filter is defined as the product of the L1 norm of its weight and its sign:
\[
q_{n,l} = (\text{sign}(W_{n,l}))\times \| W_{n,l} \|_1
\]

- **Effect of force**:
- Repulsive forces are generated between similar charges (with the same sign), causing the weights of the filters to approach zero.
- Attractive forces are generated between different charges (with different signs), causing the weights of the filters to remain non - zero values.

In this way, unimportant filters can be effectively pruned while important information is retained, thereby reducing model complexity without significantly degrading model performance.

### Experimental results

The author verified the effectiveness of this method on the MNIST, CIFAR, and ImageNet datasets. The results show that its performance is comparable to or even better than that of the existing structured pruning methods, while having a simpler implementation and less need for fine - tuning.