Abstract:We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level. By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection. In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection. CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.

What problem does this paper attempt to address?

The paper aims to address the issue of uncertainty handling in Intent Classification (IC) within task-oriented dialogue systems, especially when user input may be incomplete, erroneous, or contain linguistic ambiguities. The paper proposes a framework called "Conformal Intent Classification and Clarification" (CICC) for quickly and accurately performing intent classification and generating Clarification Questions (CQs) when necessary to enhance the user experience of the dialogue system. Key contributions of the CICC framework include: 1. **Decision Mechanism**: Determining when to ask clarification questions, the specific information content of the questions, and how to formulate the questions. 2. **Statistical Guarantee**: Using conformal prediction methods to convert the prediction uncertainty of any intent classifier into a prediction set that includes the true intent, with statistical guarantees at a predefined confidence level. 3. **Handling Model Uncertainty**: Capable of handling model uncertainty, avoiding the generation of clarification questions when the model is too uncertain, thereby reducing user confusion. 4. **Small-Scale Clarification Questions**: The generated clarification questions are small in scale, helping to quickly and accurately resolve user queries. 5. **Out-of-Scope Detection**: The framework can be extended to identify user inputs that are outside the predefined intent scope (i.e., out-of-scope inputs). Experimental results show that CICC, when combined with three different intent classification models on seven intent recognition datasets, outperforms heuristic-based baseline methods in all cases, particularly in handling ambiguous inputs. Additionally, CICC demonstrates the ability to address out-of-scope detection, even though this is typically considered a different challenge from handling model uncertainty.

Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition

Class Lifelong Learning for Intent Detection via Structure Consolidation Networks

Resolving Intent Ambiguities by Retrieving Discriminative Clarifying Questions

Real-time Caller Intent Detection In Human-Human Customer Support Spoken Conversations

Effective Open Intent Classification with K-center Contrastive Learning and Adjustable Decision Boundary

Continual Few-shot Intent Detection.

Fuzzy Classification of Multi-intent Utterances

IFCRL: Interval-intent Fuzzy Concept Re-cognition Learning Model

Conversational Factor Information Retrieval Model (ConFIRM)

Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinement

Multi-Stage Coarse-to-Fine Contrastive Learning for Conversation Intent Induction

Few‐Shot Contrastive Learning‐Based Multi‐Round Dialogue Intent Classification Method

Asking Clarifying Questions Based on Negative Feedback in Conversational Search

Unknown Intention Detection Method Based on Dynamic Constraint Boundary

Intent-Aware Dialogue Generation and Multi-Task Contrastive Learning for Multi-Turn Intent Classification

A Prompt Learning Based Intent Recognition Method on a Chinese Implicit Intent Dataset CIID

An intent recognition pipeline for conversational AI

Calibrate and Refine! A Novel and Agile Framework for ASR-error Robust Intent Detection

Conditioning to true content and artificial intelligence in psychophysiological intention recognition

Improved Out-of-Scope Intent Classification with Dual Encoding and Threshold-based Re-Classification

Intuitionistic Fuzzy Cognitive Maps for Interpretable Image Classification