How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?

Ryan Liu,Theodore R. Sumers,Ishita Dasgupta,Thomas L. Griffiths

2024-02-13

Abstract:In day-to-day communication, people often approximate the truth - for example, rounding the time or omitting details - in order to be maximally helpful to the listener. How do large language models (LLMs) handle such nuanced trade-offs? To address this question, we use psychological models and experiments designed to characterize human behavior to analyze LLMs. We test a range of LLMs and explore how optimization for human preferences or inference-time reasoning affects these trade-offs. We find that reinforcement learning from human feedback improves both honesty and helpfulness, while chain-of-thought prompting skews LLMs towards helpfulness over honesty. Finally, GPT-4 Turbo demonstrates human-like response patterns including sensitivity to the conversational framing and listener's decision context. Our findings reveal the conversational values internalized by LLMs and suggest that even these abstract values can, to a degree, be steered by zero-shot prompting.

Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper explores how large language models (LLMs) balance between honesty and usefulness. Specifically:

1. **Research Background**:
- In everyday conversations, people often approximate facts to maximize help to the listener, such as rounding off time, distance, or monetary values.
- How do large language models handle this trade-off between honesty and usefulness when generating language?

2. **Research Objectives**:
- Establish a formal model to describe usefulness and formalize it based on cognitive science theories.
- Propose a framework to test the value trade-offs made by LLMs in ambiguous conversational contexts.
- Investigate how model training (through Reinforcement Learning from Human Feedback, RLHF), prompting (Chain-of-Thought prompting, CoT), and question framing (realistic or abstract) affect these trade-offs.

3. **Key Findings**:
- RLHF significantly enhances both honesty and usefulness.
- Chain-of-Thought prompting (CoT) improves usefulness but may reduce honesty.
- The latest GPT-4 Turbo model exhibits value trade-off patterns similar to humans and can be adjusted through simple prompts.

Through these studies, the paper reveals the current behavior patterns of LLMs in balancing honesty and usefulness and suggests directions for improvement.