Abstract:Existing preference optimization methods are mainly designed for directly learning from human feedback with the assumption that paired examples (preferred vs. dis-preferred) are available. In contrast, we propose a method that can leverage unpaired preferred or dis-preferred examples, and works even when only one type of feedback (positive or negative) is available. This flexibility allows us to apply it in scenarios with varying forms of feedback and models, including training generative language models based on human feedback as well as training policies for sequential decision-making problems, where learned (value) functions are available. Our approach builds upon the probabilistic framework introduced in (Dayan and Hinton, 1997), which proposes to use expectation-maximization (EM) to directly optimize the probability of preferred outcomes (as opposed to classic expected reward maximization). To obtain a practical algorithm, we identify and address a key limitation in current EM-based methods: when applied to preference optimization, they solely maximize the likelihood of preferred examples, while neglecting dis-preferred samples. We show how one can extend EM algorithms to explicitly incorporate dis-preferred outcomes, leading to a novel, theoretically grounded, preference optimization algorithm that offers an intuitive and versatile way to learn from both positive and negative feedback.

Optimality, Belief and Preference

Preference, Priorities and Belief

Belief-Based Preference

The Logic of Belief Preference: from Single Subject to Multiple Subjects

From Incomplete Preferences to Ranking via Optimization

Entanglement of Preference, Knowledge and Belief

Approaches for Mapping Between Preferential Probabilities and Relative Design Preference Ratings

Preference Optimization as Probabilistic Inference

Preference over Worlds: Static Logic

Preference from Priorities: Dynamic Logic

Minimal Preference Change

A Two-Level Perspective on Preference

Conditions on Preference Relations that Guarantee the Existence of Optimal Policies

Deontic Logic And Preference Change

Sequential Preference-Based Optimization

Preference Change: a Quantitative Approach

Preferences Implicit in the State of the World

Preference logic, conditionals and solution concepts in games

Preference over Worlds: Dynamic Logic

Coarse Descriptions and Cautious Preferences

Dynamic Logic of Preference Upgrade.