Abstract:Multimodal intent recognition aims to leverage diverse modalities such as expressions, body movements and tone of speech to comprehend user's intent, constituting a critical task for understanding human language and behavior in real-world multimodal scenarios. Nevertheless, the majority of existing methods ignore potential correlations among different modalities and own limitations in effectively learning semantic features from nonverbal modalities. In this paper, we introduce a token-level contrastive learning method with modality-aware prompting (TCL-MAP) to address the above challenges. To establish an optimal multimodal semantic environment for text modality, we develop a modality-aware prompting module (MAP), which effectively aligns and fuses features from text, video and audio modalities with similarity-based modality alignment and cross-modality attention mechanism. Based on the modality-aware prompt and ground truth labels, the proposed token-level contrastive learning framework (TCL) constructs augmented samples and employs NT-Xent loss on the label token. Specifically, TCL capitalizes on the optimal textual semantic insights derived from intent labels to guide the learning processes of other modalities in return. Extensive experiments show that our method achieves remarkable improvements compared to state-of-the-art methods. Additionally, ablation analyses demonstrate the superiority of the modality-aware prompt over the handcrafted prompt, which holds substantial significance for multimodal prompt learning. The codes are released at <a class="link-external link-https" href="https://github.com/thuiar/TCL-MAP" rel="external noopener nofollow">this https URL</a>.

Prompt Learning for Multimodal Intent Recognition with Modal Alignment Perception

Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Modality-invariant and Specific Prompting for Multimodal Human Perception Understanding

Deep Correlated Prompting for Visual Recognition with Missing Modalities

MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modality

Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition

Multimodal Prompting with Missing Modalities for Visual Recognition

ModalPrompt:Dual-Modality Guided Prompt for Continual Learning of Large Multimodal Models

Tuning Multi-mode Token-level Prompt Alignment across Modalities

An Effective Multimodal Representation and Fusion Method for Multimodal Intent Recognition

Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements

POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models

Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning

Memory-Inspired Temporal Prompt Interaction for Text-Image Classification

Multi-Prompt with Depth Partitioned Cross-Modal Learning

MaPLe: Multi-modal Prompt Learning

Towards Robust Multimodal Prompting With Missing Modalities

Prompt Link Multimodal Fusion in Multimodal Sentiment Analysis

Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting

Adaptive Multi-Modality Prompt Learning