Abstract:Automatic open-domain dialogue evaluation has attracted increasing attention. Trainable evaluation metrics, typically trained with true positive and randomly selected negative responses, tend to assign higher scores to responses that share greater content similarity with a given context. However, adversarial negative responses, despite possessing high content similarity with the contexts, are semantically different. Consequently, existing evaluation metrics are not robust enough to evaluate such responses, resulting in low correlations with human judgments. While recent studies have demonstrated the effectiveness of Large Language Models (LLMs) for open-domain dialogue evaluation, they still face challenges in effectively handling adversarial negative examples. In this paper, we propose an effective framework for open-domain dialogue evaluation, which combines domain-specific language models (SLMs) enhanced with Abstract Meaning Representation (AMR) knowledge with LLMs. The SLMs can explicitly incorporate AMR graph information of the dialogue through a gating mechanism for enhanced dialogue semantic representation learning. Both the evaluation result from the SLMs and the AMR graph information are incorporated into the LLM's prompt for enhanced evaluation performance. Experimental results on open-domain dialogue evaluation tasks demonstrate the superiority of our method compared to a wide range of state-of-the-art baselines, especially in discriminating adversarial negative responses. Our code and data are publicly available at <a class="link-external link-https" href="https://github.com/Bernard-Yang/SIMAMR" rel="external noopener nofollow">this https URL</a>.

Overview of SMP-CAIL2020-Argmine: the Interactive Argument-Pair Extraction in Judgement Document Challenge

Overview of the CAIL 2023 Argument Mining Track

Overview of Argumentative Text Understanding for AI Debater Challenge

MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation

Which Side Are You On? A Multi-task Dataset for End-to-End Argument Summarisation and Evaluation

IAM: A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks

Overview of CAIL2018: Legal Judgment Prediction Competition.

Assessing Open-Source Large Language Models on Argumentation Mining Subtasks

CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain

Overview of AI-Debater 2023: The Challenges of Argument Generation Tasks

Overview of ImageArg-2023: The First Shared Task in Multimodal Argument Mining

RuArg-2022: Argument Mining Evaluation

CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction.

A Two-Stream AMR-enhanced Model for Document-level Event Argument Extraction

AntCritic: Argument Mining for Free-Form and Visually-Rich Financial Comments

Multilingual Argument Mining: Datasets and Analysis

Exploring the Potential of Large Language Models in Computational Argumentation

ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges

CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structures

Emphasising Structured Information: Integrating Abstract Meaning Representation into LLMs for Enhanced Open-Domain Dialogue Evaluation

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration