Abstract:Multimodal intent recognition aims to leverage diverse modalities such as expressions, body movements and tone of speech to comprehend user's intent, constituting a critical task for understanding human language and behavior in real-world multimodal scenarios. Nevertheless, the majority of existing methods ignore potential correlations among different modalities and own limitations in effectively learning semantic features from nonverbal modalities. In this paper, we introduce a token-level contrastive learning method with modality-aware prompting (TCL-MAP) to address the above challenges. To establish an optimal multimodal semantic environment for text modality, we develop a modality-aware prompting module (MAP), which effectively aligns and fuses features from text, video and audio modalities with similarity-based modality alignment and cross-modality attention mechanism. Based on the modality-aware prompt and ground truth labels, the proposed token-level contrastive learning framework (TCL) constructs augmented samples and employs NT-Xent loss on the label token. Specifically, TCL capitalizes on the optimal textual semantic insights derived from intent labels to guide the learning processes of other modalities in return. Extensive experiments show that our method achieves remarkable improvements compared to state-of-the-art methods. Additionally, ablation analyses demonstrate the superiority of the modality-aware prompt over the handcrafted prompt, which holds substantial significance for multimodal prompt learning. The codes are released at <a class="link-external link-https" href="https://github.com/thuiar/TCL-MAP" rel="external noopener nofollow">this https URL</a>.

A Prompt Learning Based Intent Recognition Method on a Chinese Implicit Intent Dataset CIID

Research on Implicit Intent Recognition Method Based on Prompt Learning

Class Lifelong Learning for Intent Detection via Structure Consolidation Networks

Similarity Learning with Implicit-Network and Explicit-Network for Zero-Shot Intent Detection

A Multi-Concept Semantic Representation System for Chinese Intent Recognition

Domain-Enhanced Prompt Learning for Chinese Implicit Hate Speech Detection

Learning Class-Transductive Intent Representations for Zero-shot Intent Detection

Episode-Based Prompt Learning for Any-Shot Intent Detection.

Prompt Learning with Knowledge Memorizing Prototypes for Generalized Few-Shot Intent Detection

Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognition

Adaptive Prompt Learning with Distilled Connective Knowledge for Implicit Discourse Relation Recognition

Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition

Continual Generalized Intent Discovery: Marching Towards Dynamic and Open-world Intent Recognition

Continual Few-shot Intent Detection.

Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset

Promoting Unified Generative Framework with Descriptive Prompts for Joint Multi-Intent Detection and Slot Filling

Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue Generation

We Know What You Will Ask: A Dialogue System for Multi-intent Switch and Prediction

Multi-Task Deep Learning for User Intention Understanding in Speech Interaction Systems

Multi-Stage Coarse-to-Fine Contrastive Learning for Conversation Intent Induction

Contextual Information and Commonsense Based Prompt for Emotion Recognition in Conversation