Abstract:Generative pre-trained transformers (GPT) have recently demonstrated excellent performance in various natural language tasks. The development of ChatGPT and the recently released GPT-4 model has shown competence in solving complex and higher-order reasoning tasks without further training or fine-tuning. However, the applicability and strength of these models in classifying legal texts in the context of argument mining are yet to be realized and have not been tested thoroughly. In this study, we investigate the effectiveness of GPT-like models, specifically GPT-3.5 and GPT-4, for argument mining via prompting. We closely study the model's performance considering diverse prompt formulation and example selection in the prompt via semantic search using state-of-the-art embedding models from OpenAI and sentence transformers. We primarily concentrate on the argument component classification task on the legal corpus from the European Court of Human Rights. To address these models' inherent non-deterministic nature and make our result statistically sound, we conducted 5-fold cross-validation on the test set. Our experiments demonstrate, quite surprisingly, that relatively small domain-specific models outperform GPT 3.5 and GPT-4 in the F1-score for premise and conclusion classes, with 1.9% and 12% improvements, respectively. We hypothesize that the performance drop indirectly reflects the complexity of the structure in the dataset, which we verify through prompt and data analysis. Nevertheless, our results demonstrate a noteworthy variation in the performance of GPT models based on prompt formulation. We observe comparable performance between the two embedding models, with a slight improvement in the local model's ability for prompt selection. This suggests that local models are as semantically rich as the embeddings from the OpenAI model. Our results indicate that the structure of prompts significantly impacts the performance of GPT models and should be considered when designing them.

Efficient argument classification with compact language models and ChatGPT-4 refinements

In-Context Learning and Fine-Tuning GPT for Argument Mining

Exploring the Potential of Large Language Models in Computational Argumentation

Enhancing argumentation component classification using contextual language model

Can Large Language Models perform Relation-based Argument Mining?

Performance analysis of large language models in the domain of legal argument mining

Multi-Task Learning Improves Performance In Deep Argument Mining Models

Pattern-Based Meta Graph Neural Networks for Argument Classifications

Assessing Good, Bad and Ugly Arguments Generated by ChatGPT: a New Dataset, its Methodology and Associated Tasks

DeepA2: A Modular Framework for Deep Argument Analysis with Pretrained Neural Text2Text Language Models

AutoAM: An End-To-End Neural Model for Automatic and Universal Argument Mining

Aspect-Based Argument Mining

Argumentation Mining in User-Generated Web Discourse

Can Unsupervised Knowledge Transfer from Social Discussions Help Argument Mining?

Classifying Argumentative Relations Using Logical Mechanisms and Argumentation Schemes

End-to-End Argument Mining as Augmented Natural Language Generation

Active Learning for Argument Mining: A Practical Approach

Assessing Open-Source Large Language Models on Argumentation Mining Subtasks

Towards a Holistic View on Argument Quality Prediction

Multilingual Argument Mining: Datasets and Analysis

"I'd Like to Have an Argument, Please": Argumentative Reasoning in Large Language Models