Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding

Yanda Li,Dixuan Wang,Jiaqing Liang,Guochao Jiang,Qianyu He,Yanghua Xiao,Deqing Yang

2024-04-04

Abstract:Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but they still struggle with some complicated reasoning tasks including logical reasoning. One non-negligible reason for LLMs' suboptimal performance on logical reasoning is their overlooking of understanding logical fallacies correctly. To evaluate LLMs' capability of logical fallacy understanding (LFU), we propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW in this paper. Towards these LFU tasks, we have successfully constructed a new dataset LFUD based on GPT-4 accompanied by a little human effort. Our extensive experiments justify that our LFUD can be used not only to evaluate LLMs' LFU capability, but also to fine-tune LLMs to obtain significantly enhanced performance on logical reasoning.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

This paper attempts to solve the problem of poor performance of large - language models (LLMs) in logical reasoning tasks, especially their deficiencies in understanding and identifying logical fallacies. The author believes that the sub - optimal performance of LLMs in logical reasoning is partly due to their failure to correctly understand logical fallacies. Therefore, the paper proposes five specific tasks to evaluate the understanding ability of LLMs for logical fallacies and constructs a new dataset LFUD for evaluating and enhancing the logical fallacy understanding ability of LLMs. Through these tasks and datasets, researchers hope to not only evaluate the logical fallacy understanding ability of LLMs, but also significantly improve their performance in logical reasoning tasks by fine - tuning these models.

Specifically, the main contributions of the paper include:
1. Proposing five specific tasks to truly evaluate the understanding ability of LLMs for logical fallacies from three cognitive dimensions of WHAT (what it is), WHY (why) and HOW (how).
2. Designing a new framework for constructing a high - quality dataset LFUD to evaluate the logical fallacy understanding ability of LLMs and enhancing the performance of LLMs in logical reasoning through these tasks.
3. The constructed LFUD dataset contains 4,020 instances and involves 12 types of logical fallacies. Experimental results show that fine - tuning LLMs with LFUD can significantly improve their performance in logical reasoning tasks.

Through these methods, the paper aims to systematically evaluate and enhance the logical fallacy understanding ability of LLMs, thereby improving their performance in complex logical reasoning tasks.