Gemma 2: Improving Open Language Models at a Practical Size

Gemma Team,Morgane Riviere,Shreya Pathak,Pier Giuseppe Sessa,Cassidy Hardin,Surya Bhupatiraju,Léonard Hussenot,Thomas Mesnard,Bobak Shahriari,Alexandre Ramé,Johan Ferret,Peter Liu,Pouya Tafti,Abe Friesen,Michelle Casbon,Sabela Ramos,Ravin Kumar,Charline Le Lan,Sammy Jerome,Anton Tsitsulin,Nino Vieillard,Piotr Stanczyk,Sertan Girgin,Nikola Momchev,Matt Hoffman,Shantanu Thakoor,Jean-Bastien Grill,Behnam Neyshabur,Olivier Bachem,Alanna Walton,Aliaksei Severyn,Alicia Parrish,Aliya Ahmad,Allen Hutchison,Alvin Abdagic,Amanda Carl,Amy Shen,Andy Brock,Andy Coenen,Anthony Laforge,Antonia Paterson,Ben Bastian,Bilal Piot,Bo Wu,Brandon Royal,Charlie Chen,Chintu Kumar,Chris Perry,Chris Welty,Christopher A. Choquette-Choo,Danila Sinopalnikov,David Weinberger,Dimple Vijaykumar,Dominika Rogozińska,Dustin Herbison,Elisa Bandy,Emma Wang,Eric Noland,Erica Moreira,Evan Senter,Evgenii Eltyshev,Francesco Visin,Gabriel Rasskin,Gary Wei,Glenn Cameron,Gus Martins,Hadi Hashemi,Hanna Klimczak-Plucińska,Harleen Batra,Harsh Dhand,Ivan Nardini,Jacinda Mein,Jack Zhou,James Svensson,Jeff Stanway,Jetha Chan,Jin Peng Zhou,Joana Carrasqueira,Joana Iljazi,Jocelyn Becker,Joe Fernandez,Joost van Amersfoort,Josh Gordon,Josh Lipschultz,Josh Newlan,Ju-yeong Ji,Kareem Mohamed,Kartikeya Badola,Kat Black,Katie Millican,Keelin McDonell,Kelvin Nguyen,Kiranbir Sodhia,Kish Greene,Lars Lowe Sjoesund,Lauren Usui,Laurent Sifre,Lena Heuermann,Leticia Lago,Lilly McNealus,et al. (98 additional authors not shown)

2024-10-02

Abstract:In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Improving the performance of small language models**: Researchers attempt to enhance the performance of small language models (such as models with 2B and 9B parameters) through techniques like knowledge distillation, rather than merely increasing the amount of training data. By distilling knowledge from larger teacher models, the performance of smaller models can be significantly improved.

2. **Optimizing model architecture**: The paper introduces the Gemma 2 model family, including different versions with 2B, 9B, and 27B parameters. These models employ a combination of local sliding window attention and global attention, as well as a grouped-query attention mechanism, to enhance performance while keeping the models lightweight.

3. **Enhancing multi-turn dialogue capability**: Through supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), the models' performance in multi-turn dialogues is improved, and their safety is enhanced.

4. **Reducing memorization effects**: Researchers evaluated the models' degree of memorization of training data and found that the Gemma 2 models exhibit lower rates of verbatim memorization and approximate memorization, thereby reducing the risk of privacy leakage.

Overall, the Gemma 2 models aim to significantly improve performance across various tasks while maintaining a smaller model size, and to enhance safety and reliability in practical applications through multiple techniques and optimizations.