Abstract:Assessing the quality of arguments is both valuable and challenging. Humans often find that making pairwise comparisons between a target argument and several reference arguments facilitates a more precise judgment of the target argument's quality. Inspired by this, we propose a comparison-based framework for argument quality assessments (CompAQA), which scores the quality of an argument through multiple pairwise comparisons. Additionally, we introduce an argument order-based data augmentation strategy to enhance CompAQA's relative quality comparison ability. By introducing multiple reference arguments for pairwise comparisons, CompAQA improves the objectivity and precision of argument quality assessments. Another advantage of CompAQA is its ability to integrate both pairwise argument quality classification and argument quality ranking tasks into a unified framework, distinguishing it from existing methods. We conduct extensive experiments using various pre-trained encoder-only models. Our experiments involve two argument quality ranking datasets (IBM-ArgQ-5.3kArgs and IBM-Rank-30k) and one pairwise argument quality classification dataset (IBM-ArgQ-9.1kPairs). Overall, CompAQA significantly outperforms several strong baselines. Specifically, when using the RoBERTa model as a backbone, CompAQA outperforms the previous best method on the IBM-Rank-30k dataset, improving Pearson correlation by 0.0203 and Spearman correlation by 0.0148. On the IBM-ArgQ-5.3kArgs dataset, it shows improvements of 0.0069 in Pearson correlation and 0.0208 in Spearman correlation. Furthermore, CompAQA demonstrates a 4.71% increase in accuracy over the baseline method on the IBM-ArgQ-9.1kPairs dataset. We also show that CompAQA can be effectively applied to fine-tune larger decoder-only pre-trained models, such as Llama.

A Comparison-Based Framework for Argument Quality Assessment

Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare

Argument Quality Assessment in the Age of Instruction-Following Large Language Models

Contextualizing Argument Quality Assessment with Relevant Knowledge

CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs

Rhetoric, Logic, and Dialectic: Advancing Theory-based Argument Quality Assessment in Natural Language Processing

Towards Open-ended Visual Quality Comparison

Strategy for Boosting Pair Comparison and Improving Quality Assessment Accuracy

ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges

Quality Assessment for Comparing Image Enhancement Algorithms

Revisiting the Role of Similarity and Dissimilarity in Best Counter Argument Retrieval

Pairwise Comparisons Are All You Need

Architectural Sweet Spots for Modeling Human Label Variation by the Example of Argument Quality: It's Best to Relate Perspectives!

Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale

The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators

Towards a Holistic View on Argument Quality Prediction

Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisons

Pairwise comparison and rank learning for image quality assessment

Claim Optimization in Computational Argumentation

GMC-IQA: Exploiting Global-correlation and Mean-opinion Consistency for No-reference Image Quality Assessment

Content-Diverse Comparisons improve IQA