Abstract:The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary ~5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be ~1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be ~2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.

Curb Your Carbon Emissions: Benchmarking Carbon Emissions in Machine Translation

Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training

GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost

CEGI: Measuring the trade-off between efficiency and carbon emissions for SLMs and VLMs

Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models

Reporting and Analysing the Environmental Impact of Language Models on the Example of Commonsense Question Answering with External Knowledge

Quantifying the Carbon Emissions of Machine Learning

The Ecological Footprint of Neural Machine Translation Systems

Carbon Emissions and Large Neural Network Training

Reconciling the contrasting narratives on the environmental impact of large language models

A Survey on Model Compression and Acceleration for Pretrained Language Models

Computing Within Limits: An Empirical Study of Energy Consumption in ML Training and Inference

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

Evaluating and Optimizing the Effectiveness of Neural Machine Translation in Supporting Code Retrieval Models: A Study on the CAT Benchmark

The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics

Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research

xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics

A Survey of Sustainability in Large Language Models: Applications, Economics, and Challenges

Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation

Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages

On the Evaluation of Neural Code Translation: Taxonomy and Benchmark