Abstract:Background. The rapid growth of Language Models (LMs), particularly in code generation, requires substantial computational resources, raising concerns about energy consumption and environmental impact. Optimizing LMs inference for energy efficiency is crucial, and Small Language Models (SLMs) offer a promising solution to reduce resource demands. Aim. Our goal is to analyze the impact of deep learning runtime engines and execution providers on energy consumption, execution time, and computing-resource utilization from the point of view of software engineers conducting inference in the context of code SLMs. Method. We conducted a technology-oriented, multi-stage experimental pipeline using twelve code generation SLMs to investigate energy consumption, execution time, and computing-resource utilization across the configurations. Results. Significant differences emerged across configurations. CUDA execution provider configurations outperformed CPU execution provider configurations in both energy consumption and execution time. Among the configurations, TORCH paired with CUDA demonstrated the greatest energy efficiency, achieving energy savings from 37.99% up to 89.16% compared to other serving configurations. Similarly, optimized runtime engines like ONNX with the CPU execution provider achieved from 8.98% up to 72.04% energy savings within CPU-based configurations. Also, TORCH paired with CUDA exhibited efficient computing-resource utilization. Conclusions. Serving configuration choice significantly impacts energy efficiency. While further research is needed, we recommend the above configurations best suited to software engineers' requirements for enhancing serving efficiency in energy and performance.

Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems

Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads

Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference

Analyzing the Energy and Accuracy of LLMs in Software Development

From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference

DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency

Can Large-Language Models Help us Better Understand and Teach the Development of Energy-Efficient Software?

Energy consumption of code small language models serving with runtime engines and execution providers

ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency

Efficient and Economic Large Language Model Inference with Attention Offloading

Decentralized LLM Inference over Edge Networks with Energy Harvesting

SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimization

Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models

Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference

Large Language Models for Energy-Efficient Code: Emerging Results and Future Directions

Efficient Deployment of Large Language Model Across Cloud-Device Systems

LLM-based Frameworks for Power Engineering from Routine to Novel Tasks

Towards Pareto Optimal Throughput in Small Language Model Serving

Distributed Inference Performance Optimization for LLMs on CPUs

All Language Models Large and Small

Distributed Inference and Fine-tuning of Large Language Models Over The Internet