Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluation

Federico A. Galatolo,Mario G.C.A. Cimino

DOI: https://doi.org/10.48550/arXiv.2311.15698

2023-11-27

Abstract:This study introduces a novel approach for generating high-quality, language-specific chat corpora using a self-chat mechanism. We combine a generator LLM for creating new samples and an embedder LLM to ensure diversity. A new Masked Language Modelling (MLM) model-based quality assessment metric is proposed for evaluating and filtering the corpora. Utilizing the llama2-70b as the generator and a multilingual sentence transformer as embedder, we generate an Italian chat corpus and refine the Fauno corpus, which is based on translated English ChatGPT self-chat data. The refinement uses structural assertions and Natural Language Processing techniques. Both corpora undergo a comprehensive quality evaluation using the proposed MLM model-based quality metric. The Italian LLM fine-tuned with these corpora demonstrates significantly enhanced language comprehension and question-answering skills. The resultant model, cerbero-7b, establishes a new state-of-the-art for Italian LLMs. This approach marks a substantial advancement in the development of language-specific LLMs, with a special emphasis on augmenting corpora for underrepresented languages like Italian.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the performance of large language models (LLMs) in specific languages (such as Italian), especially in dialogue generation and understanding. Specifically, the paper addresses this problem through the following methods:

1. **Generate high - quality language - specific chat corpora**: The paper presents a new method for generating high - quality language - specific chat corpora using a self - chatting mechanism. This method combines a generator LLM (for creating new samples) and an embedder LLM (for ensuring diversity).

2. **Propose new quality evaluation metrics**: To evaluate and filter the generated corpora, the paper proposes a quality evaluation metric based on the Masked Language Modelling (MLM) model. This metric quantifies the quality of a sentence by calculating the probability distribution of masked words in each sentence, using the negative log - likelihood (NLL) as the evaluation criterion.

3. **Optimize and enhance existing corpora**: The paper optimizes and enhances the existing Fauno corpus by using structural assertions and natural language processing techniques to improve its quality and diversity.

4. **Generate a new Italian chat corpus**: Using the self - chatting mechanism and a multilingual sentence embedder, a new Italian chat corpus is generated and screened and optimized by the quality evaluation metric.

5. **Model fine - tuning and evaluation**: The existing LLM is fine - tuned using the generated high - quality corpus, and the performance of the fine - tuned model is evaluated through multiple benchmark test tasks (such as SQuAD - it, EVALITA, etc.), especially its performance in language understanding and question - answering tasks.

Through these methods, the paper aims to significantly improve the language understanding and question - answering abilities of the Italian LLM, and ultimately establish a new Italian LLM - cerbero - 7b, thereby promoting the development of LLMs in specific languages, especially in under - resourced languages such as Italian.