OpenAssistant Conversations -- Democratizing Large Language Model Alignment

Andreas Köpf,Yannic Kilcher,Dimitri von Rütte,Sotiris Anagnostidis,Zhi-Rui Tam,Keith Stevens,Abdullah Barhoum,Nguyen Minh Duc,Oliver Stanley,Richárd Nagyfi,Shahul ES,Sameer Suri,David Glushkov,Arnav Dantuluri,Andrew Maguire,Christoph Schuhmann,Huu Nguyen,Alexander Mattick

2023-10-31

Abstract:Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains. However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary. In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 complete and fully annotated conversation trees. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers. Models trained on OpenAssistant Conversations show consistent improvements on standard benchmarks over respective base models. We release our code and data under a fully permissive licence.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the issue of aligning large language models (LLMs) with human preferences to improve their usability and safety. Specifically:

1. **Model Alignment Issue**: The paper points out that although current large language models have a massive number of parameters and rich training data, there is still a discrepancy between model predictions and their intended use in practical applications. This inconsistency mainly manifests in the content generated by the models, which may not align with human values, intentions, or preferences.

2. **Scarcity of High-Quality Feedback Data**: The most advanced alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), rely on high-quality human feedback data. However, creating such datasets is costly and often proprietary, limiting the research community's progress in large-scale alignment.

To address these issues, the authors released the OpenAssistant Conversations dataset, a large-scale dialogue tree dataset containing 161,443 messages, covering 35 languages, and annotated with 461,292 quality ratings. This dataset was collaboratively built by over 13,500 volunteers worldwide, aiming to promote more diverse and inclusive research through open access. This allows researchers to explore complex human language and interaction patterns, thereby better aligning large language models. Additionally, the paper discusses the composition of the dataset, the characteristics of the contributors, and the results of their satisfaction survey. Experiments were also conducted, demonstrating that models fine-tuned using this dataset outperformed baseline models in multiple benchmark tests.