Abstract:Multimodal intent understanding is a significant research area that requires effectively leveraging multiple modalities to analyze human language. Existing methods face two main challenges in this domain. Firstly, they have limitations in capturing nuanced and high-level semantics underlying complex in-distribution (ID) multimodal intents. Secondly, they exhibit poor generalization when confronted with unseen out-of-distribution (OOD) data in real-world scenarios. To address these issues, we propose a novel method for both ID classification and OOD detection (MIntOOD). We first introduce a weighted feature fusion network that models multimodal representations effectively. This network dynamically learns the importance of each modality, adapting to multimodal contexts. To develop discriminative representations that are conducive to both tasks, we synthesize pseudo-OOD data from convex combinations of ID data and engage in multimodal representation learning from both coarse-grained and fine-grained perspectives. The coarse-grained perspective focuses on distinguishing between ID and OOD binary classes, while the fine-grained perspective enhances the understanding of ID data by incorporating binary confidence scores. These scores help to gauge the difficulty of each sample, improving the classification of different ID classes. Additionally, the fine-grained perspective captures instance-level interactions between ID and OOD samples, promoting proximity among similar instances and separation from dissimilar ones. We establish baselines for three multimodal intent datasets and build an OOD benchmark. Extensive experiments on these datasets demonstrate that our method significantly improves OOD detection performance with a 3-10% increase in AUROC scores while achieving new state-of-the-art results in ID classification. The full data and codes are available at <a class="link-external link-https" href="https://github.com/thuiar/MIntOOD" rel="external noopener nofollow">this https URL</a>.

Delving into Out-of-Distribution Detection with Vision-Language Representations

TagOOD: A Novel Approach to Out-of-Distribution Detection via Vision-Language Representations and Class Center Learning

Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection

Zero-Shot In-Distribution Detection in Multi-Object Settings Using Vision-Language Foundation Models

COOD: Concept-based Zero-shot OOD Detection

General-Purpose Multi-Modal OOD Detection Framework

Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey

MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?

Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detection

MOODv2: Masked Image Modeling for Out-of-Distribution Detection

Multimodal Classification and Out-of-distribution Detection for Multimodal Intent Understanding

Language-Enhanced Latent Representations for Out-of-Distribution Detection in Autonomous Driving

CMG: A Class-Mixed Generation Approach to Out-of-Distribution Detection

Reframing the Relationship in Out-of-Distribution Detection

Out-of-Distribution Detection via Deep Multi-Comprehension Ensemble

Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detection

From Global to Local: Multi-scale Out-of-distribution Detection

A Unified Approach to Semi-Supervised Out-of-Distribution Detection

Enhancing Out-of-Distribution Detection with Multitesting-based Layer-wise Feature Fusion

Learning Multi-Manifold Embedding for Out-Of-Distribution Detection