A Comparison-Based Framework for Argument Quality Assessment
Jianzhu Bao,Bojun Jin,Yang Sun,Yice Zhang,Yuhang He,Ruifeng Xu
DOI: https://doi.org/10.3390/electronics13204088
IF: 2.9
2024-10-18
Electronics
Abstract:Assessing the quality of arguments is both valuable and challenging. Humans often find that making pairwise comparisons between a target argument and several reference arguments facilitates a more precise judgment of the target argument's quality. Inspired by this, we propose a comparison-based framework for argument quality assessments (CompAQA), which scores the quality of an argument through multiple pairwise comparisons. Additionally, we introduce an argument order-based data augmentation strategy to enhance CompAQA's relative quality comparison ability. By introducing multiple reference arguments for pairwise comparisons, CompAQA improves the objectivity and precision of argument quality assessments. Another advantage of CompAQA is its ability to integrate both pairwise argument quality classification and argument quality ranking tasks into a unified framework, distinguishing it from existing methods. We conduct extensive experiments using various pre-trained encoder-only models. Our experiments involve two argument quality ranking datasets (IBM-ArgQ-5.3kArgs and IBM-Rank-30k) and one pairwise argument quality classification dataset (IBM-ArgQ-9.1kPairs). Overall, CompAQA significantly outperforms several strong baselines. Specifically, when using the RoBERTa model as a backbone, CompAQA outperforms the previous best method on the IBM-Rank-30k dataset, improving Pearson correlation by 0.0203 and Spearman correlation by 0.0148. On the IBM-ArgQ-5.3kArgs dataset, it shows improvements of 0.0069 in Pearson correlation and 0.0208 in Spearman correlation. Furthermore, CompAQA demonstrates a 4.71% increase in accuracy over the baseline method on the IBM-ArgQ-9.1kPairs dataset. We also show that CompAQA can be effectively applied to fine-tune larger decoder-only pre-trained models, such as Llama.
engineering, electrical & electronic,computer science, information systems,physics, applied