Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcom

Shisen Yue,Siyuan Song,Xinyuan Cheng,Hai Hu

2024-08-01

Abstract:Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom $\textit{My Own Swordsman}$. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT-3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs' performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at <a class="link-external link-https" href="https://github.com/sjtu-compling/llm-pragmatics" rel="external noopener nofollow">this https URL</a>.

Computation and Language

What problem does this paper attempt to address?

The main focus of this paper is the understanding ability of large language models (LLMs) regarding conversational implicature, particularly whether these models can comprehend non-literal meanings and to what extent they can grasp such understanding. The study evaluates this by constructing a new dataset named SwordsmanImp, which is the first Chinese dataset based on multi-turn dialogues aimed at testing the understanding of conversational implicature. The dataset is sourced from dialogues in the Chinese sitcom "武林外传" and carefully selects 200 examples, each containing a sentence that violates one of Grice's maxims of cooperative principle.

The paper explores the following three specific questions:
1. How do state-of-the-art LLMs perform in interpreting Chinese conversational implicature?
2. What are the common error patterns of these models in multiple-choice tasks?
3. How do human participants evaluate the explanations generated by LLMs regarding conversational implicature?

The experimental results show that GPT-4 achieves human-level performance on this task, with an accuracy rate of 94%, while other models such as CausalLM (14B) also exhibit high accuracy (78.5%). Additionally, the paper finds no significant correlation between the performance of different models in interpreting conversational implicature and the types of Grice's maxims they violate. Finally, in experiments where models were required to generate explanatory texts and were evaluated by humans for their reasonableness and logic, GPT-4 also received the highest scores.