Abstract:General large language models (LLMs), represented by ChatGPT, have demonstrated significant potential in tasks such as code generation in software engineering. This has led to the development of specialized LLMs for software engineering, known as Code LLMs. A considerable portion of Code LLMs is derived from general LLMs through model fine-tuning. As a result, Code LLMs are often updated frequently and their performance can be influenced by the base LLMs. However, there is currently a lack of systematic investigation into Code LLMs and their performance. In this study, we conduct a comprehensive survey and analysis of the types of Code LLMs and their differences in performance compared to general LLMs. We aim to address three questions: (1) What LLMs are specifically designed for software engineering tasks, and what is the relationship between these Code LLMs? (2) Do Code LLMs really outperform general LLMs in software engineering tasks? (3) Which LLMs are more proficient in different software engineering tasks? To answer these questions, we first collect relevant literature and work from five major databases and open-source communities, resulting in 134 works for analysis. Next, we categorize the Code LLMs based on their publishers and examine their relationships with general LLMs and among themselves. Furthermore, we investigate the performance differences between general LLMs and Code LLMs in various software engineering tasks to demonstrate the impact of base models and Code LLMs. Finally, we comprehensively maintained the performance of LLMs across multiple mainstream benchmarks to identify the best-performing LLMs for each software engineering task. Our research not only assists developers of Code LLMs in choosing base models for the development of more advanced LLMs but also provides insights for practitioners to better understand key improvement directions for Code LLMs.

Evolving code with a large language model

Evolving Code with A Large Language Model

LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics

Large language models help computer programs to evolve

Large Language Models As Evolution Strategies

Enhancing Genetic Improvement Mutations Using Large Language Models

Large Language Models for Tuning Evolution Strategies

CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

Exploring the Improvement of Evolutionary Computation via Large Language Models

Towards Objective-Tailored Genetic Improvement Through Large Language Models

Automatically Generating CS Learning Materials with Large Language Models

A Survey on Large Language Models for Code Generation

When large language models meet evolutionary algorithms

Controlling the Mutation in Large Language Models for the Efficient Evolution of Algorithms

LLM Guided Evolution -- The Automation of Models Advancing Models

From Code to Play: Benchmarking Program Search for Games Using Large Language Models

Examination of Code generated by Large Language Models

A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

Where Are Large Language Models for Code Generation on GitHub?

Genomic Language Models: Opportunities and Challenges

SelfEvolve: A Code Evolution Framework via Large Language Models