Abstract:Large Language Models (LLMs) have become extremely potent instruments with exceptional capacities for comprehending and producing human-like text in a wide range of applications. However, the increasing size and complexity of LLMs present significant challenges in both training and deployment, leading to substantial computational and storage costs as well as heightened energy consumption. In this paper, we provide a review of recent advancements and research directions aimed at addressing these challenges and enhancing the efficiency of LLM-based systems. We begin by discussing algorithm-level acceleration techniques focused on optimizing LLM inference speed and resource utilization. We also explore LLM-hardware co-design strategies with a vision to improve system efficiency by tailoring hardware architectures to LLM requirements. Further, we delve into LLM-to-accelerator compilation approaches, which involve customizing hardware accelerators for efficient LLM deployment. Finally, as a case study to leverage LLMs for assisting circuit design, we examine LLM-aided design methodologies for an important task: High-Level Synthesis (HLS) functional verification, by creating a new dataset that contains a large number of buggy and bug-free codes, which can be essential for training LLMs to specialize on HLS verification and debugging. For each aspect mentioned above, we begin with a detailed background study, followed by the presentation of several novel solutions proposed to overcome specific challenges. We then outline future research directions to drive further advancements. Through these efforts, we aim to pave the way for more efficient and scalable deployment of LLMs across a diverse range of applications.

Large Language Models for Energy-Efficient Code: Emerging Results and Future Directions

Can Large-Language Models Help us Better Understand and Teach the Development of Energy-Efficient Software?

Large Language Models for Compiler Optimization

New Solutions on LLM Acceleration, Optimization, and Application

Language Models for Code Optimization: Survey, Challenges and Future Directions

Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models

LLM-based Frameworks for Power Engineering from Routine to Novel Tasks

A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama

Opportunities and Challenges of Applying Large Language Models in Building Energy Efficiency and Decarbonization Studies: An Exploratory Overview

Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead

The Efficiency Spectrum of Large Language Models: An Algorithmic Survey

Efficient Large Language Models: A Survey

Meta Large Language Model Compiler: Foundation Models of Compiler Optimization

Green My LLM: Studying the key factors affecting the energy consumption of code assistants

Analyzing the Energy and Accuracy of LLMs in Software Development

Large Language Models as Code Executors: An Exploratory Study

Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study

Generative Inference of Large Language Models in Edge Computing: An Energy Efficient Approach

Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

Effi-Code: Unleashing Code Efficiency in Language Models