BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop,Teven Le Scao,Angela Fan,Christopher Akiki,Ellie Pavlick,Suzana Ilić,Daniel Hesslow,Roman Castagné,Alexandra Sasha Luccioni,François Yvon,Matthias Gallé,Jonathan Tow,Alexander M. Rush,Stella Biderman,Albert Webson,Pawan Sasanka Ammanamanchi,Thomas Wang,Benoît Sagot,Niklas Muennighoff,Albert Villanova del Moral,Olatunji Ruwase,Rachel Bawden,Stas Bekman,Angelina McMillan-Major,Iz Beltagy,Huu Nguyen,Lucile Saulnier,Samson Tan,Pedro Ortiz Suarez,Victor Sanh,Hugo Laurençon,Yacine Jernite,Julien Launay,Margaret Mitchell,Colin Raffel,Aaron Gokaslan,Adi Simhi,Aitor Soroa,Alham Fikri Aji,Amit Alfassy,Anna Rogers,Ariel Kreisberg Nitzav,Canwen Xu,Chenghao Mou,Chris Emezue,Christopher Klamm,Colin Leong,Daniel van Strien,David Ifeoluwa Adelani,Dragomir Radev,Eduardo González Ponferrada,Efrat Levkovizh,Ethan Kim,Eyal Bar Natan,Francesco De Toni,Gérard Dupont,Germán Kruszewski,Giada Pistilli,Hady Elsahar,Hamza Benyamina,Hieu Tran,Ian Yu,Idris Abdulmumin,Isaac Johnson,Itziar Gonzalez-Dios,Javier de la Rosa,Jenny Chim,Jesse Dodge,Jian Zhu,Jonathan Chang,Jörg Frohberg,Joseph Tobing,Joydeep Bhattacharjee,Khalid Almubarak,Kimbo Chen,Kyle Lo,Leandro Von Werra,Leon Weber,Long Phan,Loubna Ben allal,Ludovic Tanguy,Manan Dey,Manuel Romero Muñoz,Maraim Masoud,María Grandury,Mario Šaško,Max Huang,Maximin Coavoux,Mayank Singh,Mike Tian-Jian Jiang,Minh Chien Vu,Mohammad A. Jauhar,Mustafa Ghaleb,Nishant Subramani,Nora Kassner,Nurulaqilla Khamis,Olivier Nguyen,Omar Espejel,Ona de Gibert,Paulo Villegas,et al. (293 additional authors not shown)
2023-06-27
Abstract:Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.
Computation and Language