Abstract:General large language models (LLMs), represented by ChatGPT, have demonstrated significant potential in tasks such as code generation in software engineering. This has led to the development of specialized LLMs for software engineering, known as Code LLMs. A considerable portion of Code LLMs is derived from general LLMs through model fine-tuning. As a result, Code LLMs are often updated frequently and their performance can be influenced by the base LLMs. However, there is currently a lack of systematic investigation into Code LLMs and their performance. In this study, we conduct a comprehensive survey and analysis of the types of Code LLMs and their differences in performance compared to general LLMs. We aim to address three questions: (1) What LLMs are specifically designed for software engineering tasks, and what is the relationship between these Code LLMs? (2) Do Code LLMs really outperform general LLMs in software engineering tasks? (3) Which LLMs are more proficient in different software engineering tasks? To answer these questions, we first collect relevant literature and work from five major databases and open-source communities, resulting in 134 works for analysis. Next, we categorize the Code LLMs based on their publishers and examine their relationships with general LLMs and among themselves. Furthermore, we investigate the performance differences between general LLMs and Code LLMs in various software engineering tasks to demonstrate the impact of base models and Code LLMs. Finally, we comprehensively maintained the performance of LLMs across multiple mainstream benchmarks to identify the best-performing LLMs for each software engineering task. Our research not only assists developers of Code LLMs in choosing base models for the development of more advanced LLMs but also provides insights for practitioners to better understand key improvement directions for Code LLMs.

A systematic study in intelligent software engineering based on Large Language Model

Large Language Models for Software Engineering: A Systematic Literature Review

A Survey on Large Language Models for Software Engineering

Large Language Models for Software Engineering: Survey and Open Problems

Towards an Understanding of Large Language Models in Software Engineering Tasks

The Current Challenges of Software Engineering in the Era of Large Language Models

Software Development in the Age of Intelligence: Embracing Large Language Models with the Right Approach

Large Language Model-Based Agents for Software Engineering: A Survey

Software Testing with Large Language Models: Survey, Landscape, and Vision

A Systematic Survey on Large Language Models for Algorithm Design

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future

An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

Software Service Engineering in the Era of Large Language Models

A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks

A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends

Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead

Impact of Large Language Models on Generating Software Specifications

LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead

Exploring the Potential of Large Language Models in Self-adaptive Systems