Abstract:Modern AI practices all strive towards the same goal: better results. In the context of deep learning, the term "results" often refers to the achieved accuracy on a competitive problem set. In this paper, we adopt an idea from the emerging field of Green AI to consider energy consumption as a metric of equal importance to accuracy and to reduce any irrelevant tasks or energy usage. We examine the training stage of the deep learning pipeline from a sustainability perspective, through the study of hyperparameter tuning strategies and the model complexity, two factors vastly impacting the overall pipeline's energy consumption. First, we investigate the effectiveness of grid search, random search and Bayesian optimisation during hyperparameter tuning, and we find that Bayesian optimisation significantly dominates the other strategies. Furthermore, we analyse the architecture of convolutional neural networks with the energy consumption of three prominent layer types: convolutional, linear and ReLU layers. The results show that convolutional layers are the most computationally expensive by a strong margin. Additionally, we observe diminishing returns in accuracy for more energy-hungry models. The overall energy consumption of training can be halved by reducing the network complexity. In conclusion, we highlight innovative and promising energy-efficient practices for training deep learning models. To expand the application of Green AI, we advocate for a shift in the design of deep learning models, by considering the trade-off between energy efficiency and accuracy.

What problem does this paper attempt to address?

### Problems the paper attempts to solve The paper "Uncovering Energy - Efficient Practices in Deep Learning Training: Preliminary Steps Towards Green AI" aims to explore and propose methods to achieve energy efficiency in the deep learning training process. Specifically, the paper focuses on the following aspects: 1. **Balance between energy consumption and accuracy**: - The paper proposes a new research direction, namely "Green AI", taking energy consumption as an evaluation metric as important as accuracy. - The author hopes to improve the energy efficiency of deep learning models by reducing irrelevant tasks or energy use. 2. **Energy efficiency of hyper - parameter tuning strategies**: - Three common hyper - parameter tuning strategies are studied: Grid Search, Random Search, and Bayesian Optimization. - Through experiments, the performance of these three strategies on different network architectures is compared to find the most energy - efficient strategy. 3. **Impact of network architecture on energy consumption**: - The energy consumption of different layer types (convolutional layer, linear layer, and ReLU layer) in Convolutional Neural Networks (CNN) is analyzed. - The relationship between network complexity and energy consumption is explored, especially how to reduce network complexity to reduce energy consumption while maintaining acceptable accuracy. ### Main research questions The paper proposes two main research questions (Research Questions, RQs): 1. **RQ 1**: Among Bayesian Optimization, Random Optimization, and Grid Search, which strategy is the most energy - efficient when training neural networks? - Through experiments, the performance of these three hyper - parameter tuning strategies on different network architectures is compared to evaluate their energy efficiency. 2. **RQ 2**: Can the energy consumption of neural networks be reduced while maintaining acceptable accuracy by reducing the complexity of neural networks? - By changing the layer types and numbers in the network architecture, the impact of these changes on energy consumption and model accuracy is analyzed. ### Experimental design To answer the above research questions, the paper designs two sets of experiments: 1. **Comparison of hyper - parameter tuning strategies**: - Experiments are carried out using three different neural network architectures (DenseLinearNN, DensePolyNN, and SimpleCNN). - Each strategy is optimized for multiple rounds on each network, and the highest accuracy rate and average energy consumption of each round are recorded. 2. **Impact of network architecture on energy consumption**: - By changing the number of linear layers, convolutional layers, and ReLU layers in the network, the impact of these changes on energy consumption is analyzed. - Experiments are carried out using the Random Optimization strategy, and the energy consumption and model accuracy under different configurations are recorded. ### Results 1. **Comparison of hyper - parameter tuning strategies**: - The Bayesian Optimization strategy performs best in most cases and can reach the highest accuracy rate with fewer optimization rounds. - The performance of Random Search and Grid Search is relatively close, but in some cases, Random Search is slightly better than Grid Search. 2. **Impact of network architecture on energy consumption**: - The convolutional layer is the most energy - consuming layer type, while the linear layer and ReLU layer have relatively low energy consumption. - Reducing network complexity can significantly reduce energy consumption while maintaining high model accuracy. ### Conclusion The paper emphasizes the importance of considering energy efficiency when designing deep learning models and proposes some innovative energy - saving practices. By reducing network complexity and choosing appropriate hyper - parameter tuning strategies, energy consumption can be significantly reduced without significantly sacrificing model performance. These findings are of great significance for promoting the development of Green AI.

Uncovering Energy-Efficient Practices in Deep Learning Training: Preliminary Steps Towards Green AI

A Survey on Green Deep Learning

Survey on Energy-Efficient Deep Neural Networks for Computer Vision

Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle

Data-Centric Green AI: An Exploratory Empirical Study

Toward Green AI: A Methodological Survey of the Scientific Literature

Green Edge AI: A Contemporary Survey

How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation models

A systematic review of Green AI

Measuring the Energy Consumption and Efficiency of Deep Neural Networks: An Empirical Analysis and Design Recommendations

Unveiling Energy Efficiency in Deep Learning: Measurement, Prediction, and Scoring across Edge Devices

A Green(er) World for A.I

On-Device Deep Learning: Survey on Techniques Improving Energy Efficiency of DNNs

Double-Exponential Increases in Inference Energy: The Cost of the Race for Accuracy

Computing Within Limits: An Empirical Study of Energy Consumption in ML Training and Inference

Beyond Efficiency: Scaling AI Sustainably

Energy Efficiency of Training Neural Network Architectures: An Empirical Study

Towards Green AI: Current status and future research

Watt For What: Rethinking Deep Learning's Energy-Performance Relationship

Application of AI in Optimizing Energy and Resource Management: Effectiveness of Deep Learning Models

Training Green AI Models Using Elite Samples