DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence

Daya Guo,Qihao Zhu,Dejian Yang,Zhenda Xie,Kai Dong,Wentao Zhang,Guanting Chen,Xiao Bi,Y. Wu,Y.K. Li,Fuli Luo,Yingfei Xiong,Wenfeng Liang

2024-01-26

Abstract:The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.

Software Engineering,Computation and Language,Machine Learning

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address the performance gap in code intelligence within the software development field using Large Language Models (LLMs). Specifically:

1. **Performance gap between open-source models and closed-source models**:
- The application of large language models in software development has significantly enhanced code intelligence, but most high-performance models are currently closed-source, limiting extensive research and development efforts.
- Although closed-source models are powerful, their proprietary nature prevents many researchers and developers from accessing these models.

2. **Providing high-quality open-source code models**:
- To bridge this gap, the authors propose the **DeepSeek-Coder** series of models, which include models of various scales (ranging from 1.3B to 33B parameters). These models have undergone extensive training and can perform excellently across multiple benchmarks, surpassing existing closed-source models such as Codex and GPT-3.5.

Through these efforts, the paper hopes to provide a high-performance and open-source solution to promote further research and development in the software development field.