Abstract:While deep neural networks are highly effective at solving complex tasks, large pre-trained models are commonly employed even to solve consistently simpler downstream tasks, which do not necessarily require a large model's complexity. Motivated by the awareness of the ever-growing AI environmental impact, we propose an efficiency strategy that leverages prior knowledge transferred by large models. Simple but effective, we propose a method relying on an Entropy-bASed Importance mEtRic (EASIER) to reduce the depth of over-parametrized deep neural networks, which alleviates their computational burden. We assess the effectiveness of our method on traditional image classification setups. Our code is available at <a class="link-external link-https" href="https://github.com/VGCQ/EASIER" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to reduce the number of layers in deep neural networks while maintaining performance, in order to reduce the computational burden and environmental impact**. Specifically, the authors note that although large pre - trained models are very effective in solving complex tasks, the complexity of these large models is often unnecessary when dealing with some relatively simple downstream tasks. Considering the increasing environmental impact of AI, the authors propose an entropy - based importance measurement method (EASIER) to reduce the number of layers in over - parameterized deep neural networks, thereby reducing their computational burden. This method can not only reduce the depth of the model, but also maintain the performance of the model to a certain extent. ### Main problems of the paper 1. **Resource consumption of large models**: - Large pre - trained models contain millions or even billions of parameters, resulting in billions or even trillions of floating - point operations (FLOPs) required for a single inference. - These models require huge hardware resources and energy consumption during training and deployment, which pose challenges to real - time applications and device - side applications, and have a negative impact on the environment. 2. **Limitations of existing compression techniques**: - Although there are already some compression techniques that can reduce the number of parameters in the model or reduce the numerical precision, few methods can effectively reduce the number of layers in neural networks. - Existing methods such as knowledge distillation can implicitly reduce the number of layers, but cannot guarantee that the performance will not decline. ### Solutions To solve the above problems, the authors propose the EASIER method, which is implemented through the following steps: - **Entropy - based importance measurement**: EASIER evaluates the importance of each layer of neurons by calculating their entropy, and then determines which layers can be linearized (i.e., the non - linear activation function is removed). - **Iterative optimization**: EASIER iteratively trains, evaluates, and estimates entropy, and gradually linearizes the layers with the lowest entropy until the performance drops significantly. ### Experimental results The authors conducted experiments on multiple architectures and datasets to verify the effectiveness of EASIER. The experimental results show that EASIER can maintain or even improve the performance of the model while reducing the number of layers, especially on datasets such as CIFAR - 10. ### Summary The core problem of this paper is to reduce the computational burden and environmental impact by reducing the number of layers in neural networks while maintaining the performance of the model as much as possible. The EASIER method provides an effective solution to achieve this goal through entropy - based importance measurement.

The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth

Improved Model Compression Method Based on Information Entropy

NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer

Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment

Importance estimate of features via analysis of their weight and gradient profile

Entropic Score metric: Decoupling Topology and Size in Training-free NAS

RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively

Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training

Energy Efficiency of Training Neural Network Architectures: An Empirical Study

DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks

Growing Deep Neural Network Considering with Similarity between Neurons

Reducing Data Bottlenecks in Distributed, Heterogeneous Neural Networks

Minimizing Energy Consumption of Deep Learning Models by Energy-Aware Training

Mixed Precision Neural Architecture Search for Energy Efficient Deep Learning.

Achieving Occam's razor: Deep learning for optimal model reduction

Can we avoid Double Descent in Deep Neural Networks?

Small Contributions, Small Networks: Efficient Neural Network Pruning Based on Relative Importance

Efficient and Flexible Method for Reducing Moderate-size Deep Neural Networks with Condensation

Analysis & Computational Complexity Reduction of Monocular and Stereo Depth Estimation Techniques

Problem-dependent attention and effort in neural networks with applications to image resolution and model selection