Abstract:This paper studies post-training large language models (LLMs) using preference feedback from a powerful oracle to help a model iteratively improve over itself. The typical approach for post-training LLMs involves Reinforcement Learning from Human Feedback (RLHF), which traditionally separates reward learning and subsequent policy optimization. However, such a reward maximization approach is limited by the nature of "point-wise" rewards (such as Bradley-Terry model), which fails to express complex intransitive or cyclic preference relations. While advances on RLHF show reward learning and policy optimization can be merged into a single contrastive objective for stability, they yet still remain tethered to the reward maximization framework. Recently, a new wave of research sidesteps the reward maximization presumptions in favor of directly optimizing over "pair-wise" or general preferences. In this paper, we introduce Direct Nash Optimization (DNO), a provable and scalable algorithm that marries the simplicity and stability of contrastive learning with theoretical generality from optimizing general preferences. Because DNO is a batched on-policy algorithm using a regression-based objective, its implementation is straightforward and efficient. Moreover, DNO enjoys monotonic improvement across iterations that help it improve even over a strong teacher (such as GPT-4). In our experiments, a resulting 7B parameter Orca-2.5 model aligned by DNO achieves the state-of-the-art win-rate against GPT-4-Turbo of 33% on AlpacaEval 2.0 (even after controlling for response length), an absolute gain of 26% (7% to 33%) over the initializing model. It outperforms models with far more parameters, including Mistral Large, Self-Rewarding LM (70B parameters), and older versions of GPT-4.

The Importance of Directional Feedback for LLM-based Optimizers

LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning

Exploring the True Potential: Evaluating the Black-box Optimization Capability of Large Language Models

Towards Optimizing with Large Language Models

Large Language Models as Optimizers

LLMs are Highly-Constrained Biophysical Sequence Optimizers

Can LLMs plan paths with extra hints from solvers?

When Large Language Model Meets Optimization

Large Language Models to Enhance Bayesian Optimization

Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithms

Solving General Natural-Language-Description Optimization Problems with Large Language Models

Optimizing Large Language Models for Dynamic Constraints through Human-in-the-Loop Discriminators

Large Language Model-Based Evolutionary Optimizer: Reasoning with elitism

Are Large Language Models Good Prompt Optimizers?

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

Position: Leverage Foundational Models for Black-Box Optimization

EVOLvE: Evaluating and Optimizing LLMs For Exploration

On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback

Large Language Models for Tuning Evolution Strategies