Code Llama: Open Foundation Models for Code

Baptiste Rozière,Jonas Gehring,Fabian Gloeckle,Sten Sootla,Itai Gat,Xiaoqing Ellen Tan,Yossi Adi,Jingyu Liu,Romain Sauvestre,Tal Remez,Jérémy Rapin,Artyom Kozhevnikov,Ivan Evtimov,Joanna Bitton,Manish Bhatt,Cristian Canton Ferrer,Aaron Grattafiori,Wenhan Xiong,Alexandre Défossez,Jade Copet,Faisal Azhar,Hugo Touvron,Louis Martin,Nicolas Usunier,Thomas Scialom,Gabriel Synnaeve

2024-02-01

Abstract:We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B, 34B and 70B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B, 13B and 70B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 67% and 65% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to develop a powerful, open - source code generation and completion large - language model (LLM) to improve the automation level of programming tasks. Specifically, Code Llama aims to provide the following improvements:

1. **Code Generation and Completion Ability**: By being trained on a large amount of code data, Code Llama can generate and complete code snippets more accurately and supports multiple programming languages, especially Python.

2. **Context Understanding and Filling**: Code Llama can not only complete code - completion tasks, but also fill in the missing code parts (infilling) given the surrounding context. This is especially important for applications in real - time code editors.

3. **Long - Input - Context Support**: Traditional LLMs can usually only handle shorter input sequences, while Code Llama, through improved positional embedding methods (such as RoPE), can handle input sequences up to 100,000 tokens, thus better supporting project - level code reasoning and synthesis.

4. **Instruction - Following Ability**: The Code Llama - Instruct version has been specifically fine - tuned with instructions and can better understand and execute the programming instructions given by humans, ensuring that the generated code is as expected, safe and reliable.

5. **Multilingual Support**: Besides Python, Code Llama also shows excellent performance in other programming languages (such as C++, Java, PHP, TypeScript, C# and Bash), further expanding its application scenarios.

6. **Open - Source License**: To promote research and commercial applications, Code Llama adopts a permissive license agreement, allowing for wide - spread use.

In conclusion, Code Llama aims to provide developers with an efficient, multi - functional and easy - to - use tool to simplify programming tasks and improve development efficiency.