Abstract:Alignment approaches such as RLHF and DPO are actively investigated to align large language models (LLMs) with human preferences. Commercial large language models (LLMs) like GPT-4 have been recently employed to evaluate and compare different LLM alignment approaches. These models act as surrogates for human evaluators due to their promising abilities to approximate human preferences with remarkably faster feedback and lower costs. This methodology is referred to as LLM-as-a-judge. However, concerns regarding its reliability have emerged, attributed to LLM judges' biases and inconsistent decision-making. Previous research has sought to develop robust evaluation frameworks for assessing the reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address the internal inconsistency of LLMs. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-judge methods, which leads to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM judges on alignment tasks (e.g. summarization) by defining evaluation metrics with improved theoretical interpretability and disentangling reliability metrics with LLM internal inconsistency. We develop a framework to evaluate, compare, and visualize the reliability and alignment of LLM judges to provide informative observations that help choose LLM judges for alignment tasks. Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.

Understanding The Effect Of Temperature On Alignment With Human Opinions

Aligning Language Models to User Opinions

Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models

Understanding the Learning Dynamics of Alignment with Human Feedback

Dissecting Human and LLM Preferences

Whose Opinions Do Language Models Reflect?

The Effect of Sampling Temperature on Problem Solving in Large Language Models

Using LLMs to Model the Beliefs and Preferences of Targeted Populations

Do LLMs exhibit human-like response biases? A case study in survey design

The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation

Unintended Impacts of LLM Alignment on Global Representation

Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting

Aligning language models with human preferences

Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?

Whose Emotions and Moral Sentiments Do Language Models Reflect?

Exploring Temperature Effects on Large Language Models Across Various Clinical Tasks

How Reliable Is Human Feedback For Aligning Large Language Models?

Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensiveness

Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback