Abstract:Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks, known as Code LLMs, particularly in code generation that generates source code with LLM from natural language descriptions. This burgeoning field has captured significant interest from both academic researchers and industry professionals due to its practical significance in software development, e.g., GitHub Copilot. Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of natural language processing (NLP) or software engineering (SE) or both, there is a noticeable absence of a comprehensive and up-to-date literature review dedicated to LLM for code generation. In this survey, we aim to bridge this gap by providing a systematic literature review that serves as a valuable reference for researchers investigating the cutting-edge progress in LLMs for code generation. We introduce a taxonomy to categorize and discuss the recent developments in LLMs for code generation, covering aspects such as data curation, latest advances, performance evaluation, and real-world applications. In addition, we present a historical overview of the evolution of LLMs for code generation and offer an empirical comparison using the widely recognized HumanEval and MBPP benchmarks to highlight the progressive enhancements in LLM capabilities for code generation. We identify critical challenges and promising opportunities regarding the gap between academia and practical development. Furthermore, we have established a dedicated resource website (<a class="link-external link-https" href="https://codellm.github.io" rel="external noopener nofollow">this https URL</a>) to continuously document and disseminate the most recent advances in the field.

Generative large language models in engineering design: opportunities and challenges

How Can Large Language Models Help Humans in Design and Manufacturing?

Opportunities for Large Language Models and Discourse in Engineering Design

On the use of Large Language Models in Model-Driven Engineering

Towards an Understanding of Large Language Models in Software Engineering Tasks

Large Language Models and Knowledge Graphs: Opportunities and Challenges

Large Language Models in Computer Science Education: A Systematic Literature Review

Exploring the Capabilities of Large Language Models for Generating Diverse Design Solutions

Towards Controllable Generative Design: A Conceptual Design Generation Approach Leveraging the FBS Ontology and Large Language Models

LLMs for science: Usage for code generation and data analysis

CAD-LLM: Large Language Model for CAD Generation

Large Language Models for Software Engineering: Survey and Open Problems

The Systems Engineering Approach in Times of Large Language Models

Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives

A Survey on Large Language Models for Code Generation

Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation

LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automation

Materials science in the era of large language models: a perspective

An Interdisciplinary Outlook on Large Language Models for Scientific Research

Large Language Models for Manufacturing

On the Creativity of Large Language Models