Ge Zhang,Scott Qu,Jiaheng Liu,Chenchen Zhang,Chenghua Lin,Chou Leuang Yu,Danny Pan,Esther Cheng,Jie Liu,Qunshu Lin,Raven Yuan,Tuney Zheng,Wei Pang,Xinrun Du,Yiming Liang,Yinghao Ma,Yizhi Li,Ziyang Ma,Bill Lin,Emmanouil Benetos,Huan Yang,Junting Zhou,Kaijing Ma,Minghao Liu,Morry Niu,Noah Wang,Quehry Que,Ruibo Liu,Sine Liu,Shawn Guo,Soren Gao,Wangchunshu Zhou,Xinyue Zhang,Yizhi Zhou,Yubo Wang,Yuelin Bai,Yuhan Zhang,Yuxiang Zhang,Zenith Wang,Zhenzhu Yang,Zijian Zhao,Jiajun Zhang,Wanli Ouyang,Wenhao Huang,Wenhu Chen

Abstract:Large Language Models (LLMs) have made great strides in recent years to achieve unprecedented performance across different tasks. However, due to commercial interest, the most competitive models like GPT, Gemini, and Claude have been gated behind proprietary interfaces without disclosing the training details. Recently, many institutions have open-sourced several strong LLMs like LLaMA-3, comparable to existing closed-source LLMs. However, only the model's weights are provided with most details (e.g., intermediate checkpoints, pre-training corpus, and training code, etc.) being undisclosed. To improve the transparency of LLMs, the research community has formed to open-source truly open LLMs (e.g., Pythia, Amber, OLMo), where more details (e.g., pre-training corpus and training code) are being provided. These models have greatly advanced the scientific study of these large models including their strengths, weaknesses, biases and risks. However, we observe that the existing truly open LLMs on reasoning, knowledge, and coding tasks are still inferior to existing state-of-the-art LLMs with similar model sizes. To this end, we open-source MAP-Neo, a highly capable and transparent bilingual language model with 7B parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the first fully open-sourced bilingual LLM with comparable performance compared to existing state-of-the-art LLMs. Moreover, we open-source all details to reproduce our MAP-Neo, where the cleaned pre-training corpus, data cleaning pipeline, checkpoints, and well-optimized training/evaluation framework are provided. Finally, we hope our MAP-Neo will enhance and strengthen the open research community and inspire more innovations and creativities to facilitate the further improvements of LLMs.

Meltemi: The first open Large Language Model for Greek

The Large Language Model GreekLegalRoBERTa

MELA: Multilingual Evaluation of Linguistic Acceptability

Exploring Large Language Models for Classical Philology

GREEK-BERT: The Greeks visiting Sesame Street

Large Malaysian Language Model Based on Mistral for Enhanced Local Language Understanding

BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains

Severity Prediction in Mental Health: LLM-based Creation, Analysis, Evaluation of a Novel Multilingual Dataset

A Survey on Sentiment Analysis and Opinion Mining in Greek Social Media

Lisbon Computational Linguists at SemEval-2024 Task 2: Using A Mistral 7B Model and Data Augmentation

MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series

The Greek podcast corpus: Competitive speech models for low-resourced languages with weakly supervised data

Design and implementation of an open source Greek POS Tagger and Entity Recognizer using spaCy

Hippocrates: An Open-Source Framework for Advancing Large Language Models in Healthcare

Leveraging LLMs for Translating and Classifying Mental Health Data

MEDITRON-70B: Scaling Medical Pretraining for Large Language Models

Introducing cosmosGPT: Monolingual Training for Turkish Language Models

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training