Abstract:Learning a reward model (RM) from human preferences has been an important component in aligning large language models (LLMs). The canonical setup of learning RMs from pairwise preference data is rooted in the classic Bradley-Terry (BT) model that accepts binary feedback, i.e., the label being either Response 1 is better than Response 2, or the opposite. Such a setup inevitably discards potentially useful samples (such as "tied" between the two responses) and loses more fine-grained information (such as "slightly better"). In this paper, we propose a framework for learning RMs under ordinal feedback which generalizes the case of binary preference feedback to any arbitrary granularity. Specifically, we first identify a marginal unbiasedness condition, which generalizes the assumption of the BT model in the existing binary feedback setting. The condition validates itself via the sociological concept of the wisdom of the crowd. Under the condition, we develop a natural probability model for pairwise preference data under ordinal feedback and analyze its properties. We prove the statistical benefits of ordinal feedback in terms of reducing the Rademacher complexity compared to the case of binary feedback. The proposed learning objective and the theory also extend to hinge loss and direct policy optimization (DPO). In particular, the theoretical analysis may be of independent interest when applying to a seemingly unrelated problem of knowledge distillation to interpret the bias-variance trade-off therein. The framework also sheds light on writing guidance for human annotators. Our numerical experiments validate that fine-grained feedback leads to better reward learning for both in-distribution and out-of-distribution settings. Further experiments show that incorporating a certain proportion of samples with tied preference boosts RM learning.

On Modelling Agreement and Category Distinguishability on an Ordinal Scale

Exploring Within-Rater Category Ordering: A Simulation Study Using Adjacent-Categories Mokken Scale Analysis

A Modification of McFadden's $R^2$ for Binary and Ordinal Response Models

Confirmatory Factor Analysis of Ordinal Variables With Misspecified Models

Statistical inference for agreement between multiple raters on a binary scale

LOCALLY D-OPTIMAL DESIGNS FOR HIERARCHICAL RESPONSE EXPERIMENTS

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

The Influence of Number of Categories and Threshold Values on Fit Indices in Structural Equation Modeling with Ordered Categorical Data

On the equivalence of two mixture models for rating data

Large-scale group decision-making to facilitate inter-rater reliability of human-factors analysis for the railway system

Structured Mixture of Continuation-ratio Logits Models for Ordinal Regression

Modelling and monitoring multi-relational networks with ordinal information

Statistical models for assessing agreement for quantitative data with heterogeneous random raters and replicate measurements

An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results

Diverging Preferences: When do Annotators Disagree and do Models Know?

CASSOR: Class-Aware Sample Selection for Ordinal Regression with Noisy Labels.

A Constructive Procedure for Modeling Categorical Variables: Log-Linear and Logit Models

A mixture distribution for modelling bivariate ordinal data

Maximum entropy and quantized metric models for absolute category ratings

Assessing method agreement for paired repeated binary measurements administered by multiple raters

Comparison of paired ordinal data with mis-classification and covariates adjustment