Abstract:Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs' performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at <a class="link-external link-https" href="https://github.com/sjtu-compling/llm-pragmatics" rel="external noopener nofollow">this https URL</a>.

"Is the Pope Catholic?" Applying Chain-of-Thought Reasoning to Understanding Conversational Implicatures

Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMs

Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation

Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom

Reasoning Beyond Bias: A Study on Counterfactual Prompting and Chain of Thought Reasoning

An Analysis of the Conversational Implicature of “Little Sheldon” from the Perspective of Violation of the Cooperative Principle

GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational Reasoning

Chain-of-Thought Reasoning Without Prompting

Pragmatic inference of scalar implicature by LLMs

Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters

Three Ways to Infringe a Conversational Maxim: Grice’s Theory of Conversational Implicature as a Theory of Action

Reasoning About Pragmatics with Neural Listeners and Speakers

Revisiting Conversational Meaning * —Is It Implicature, Explicature or Impliciture?

Chain-of-Thought Augmentation with Logit Contrast for Enhanced Reasoning in Language Models

How Interpretable are Reasoning Explanations from Prompting Large Language Models?

The Role of Chain-of-Thought in Complex Vision-Language Reasoning Task

Evaluating statistical language models as pragmatic reasoners

Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition

Measuring Faithfulness in Chain-of-Thought Reasoning

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought