Abstract:Nowadays, open-source large language models like LLaMA have emerged. Recent developments have incorporated supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to align these models with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of a small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the C(onditioned)-RLFT, which regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. Through extensive experiments on three standard benchmarks, our openchat-13b fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. Moreover, we use AGIEval to validate the model generalization performance, in which only openchat-13b surpasses the base model. Finally, we conduct a series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at <a class="link-external link-https" href="https://github.com/imoneoi/openchat" rel="external noopener nofollow">this https URL</a> and <a class="link-external link-https" href="https://huggingface.co/openchat" rel="external noopener nofollow">this https URL</a>.

XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters

TCMChat: A Generative Large Language Model for Traditional Chinese Medicine

YUAN 2.0: A Large Language Model with Localized Filtering-based Attention

YuLan: An Open-source Large Language Model

Baichuan 2: Open Large-scale Language Models

A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges

Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning

SNFinLLM: Systematic and Nuanced Financial Domain Adaptation of Chinese Large Language Models

YAYI 2: Multilingual Open-Source Large Language Models

NumLLM: Numeric-Sensitive Large Language Model for Chinese Finance

No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks

Large Language Models in Finance: A Survey

CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models

Chinese Fine-Grained Financial Sentiment Analysis with Large Language Models

Xmodel-1.5: An 1B-scale Multilingual LLM

Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing

Data-Centric Financial Large Language Models

OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

ChatLaw: Open-Source Legal Large Language Model with Integrated External Knowledge Bases

Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models