Camoscio: an Italian Instruction-tuned LLaMA

Andrea Santilli,Emanuele Rodolà

2023-12-19

Abstract:In recent years Large Language Models (LLMs) have increased the state of the art on several natural language processing tasks. However, their accessibility is often limited to paid API services, posing challenges for researchers in conducting extensive investigations. On the other hand, while some open-source models have been proposed by the community, they are typically English-centric or multilingual without a specific adaptation for the Italian language. In an effort to democratize the available and open resources for the Italian language, in this paper we introduce Camoscio: a language model specifically tuned to follow users' prompts in Italian. Specifically, we finetuned the smallest variant of LLaMA (7b) with LoRA on a corpus of instruction prompts translated to Italian via ChatGPT. Results indicate that the model's zero-shot performance on various downstream tasks in Italian competes favorably with existing models specifically finetuned for those tasks. All the artifacts (code, dataset, model) are released to the community at the following url: <a class="link-external link-https" href="https://github.com/teelinsan/camoscio" rel="external noopener nofollow">this https URL</a>

Computation and Language

What problem does this paper attempt to address?

The main goal of this paper is to address the usability and performance issues of large language models (LLMs) in the Italian language environment. Specifically, the paper attempts to address the following issues:

1. **Accessibility of large language models**: Many advanced large language models are typically controlled by for-profit organizations and are only available through paid APIs, which limits researchers' ability to conduct in-depth studies on these models.

2. **Limitations of existing open-source models**: While some open-source models have been proposed as alternatives, these models are often centered around English, or although they support multiple languages, their support for Italian is poor. Additionally, these models do not perform as well on specific tasks as models trained specifically for a single language.

3. **Importance of instruction tuning**: The paper points out that merely using the standard language modeling objective (i.e., predicting the next word given the previous word) to train models is not sufficient. Further instruction tuning steps are needed to improve the model's performance on downstream tasks.

To address the above issues, the paper proposes Camoscio—a LLM specifically tuned for Italian instructions. The model is based on the smallest variant of the LLaMA model (7 billion parameters) and is fine-tuned using LoRA technology. Additionally, the paper constructs an instruction tuning dataset based on the translation of Stanford University's Alpaca dataset into Italian. Finally, the paper evaluates the zero-shot performance of the Camoscio model on tasks such as news summarization, question answering, and formality transfer in Italian, comparing it with existing models specifically trained for these tasks. Experimental results show that the zero-shot performance of the Camoscio model on these tasks is comparable to or even better than existing specialized models. All relevant resources (code, datasets, model checkpoints) are openly released to the community.