Abstract:Conditional inference on joint textual and visual clues is a multi-modal reasoning task that textual clues provide prior permutation or external knowledge, which are complementary with visual content and pivotal to deducing the correct option. Previous methods utilizing pretrained vision-language models (VLMs) have achieved impressive performances, yet they show a lack of multimodal context reasoning capability, especially for text-modal information. To address this issue, we propose a Multi-modal Context Reasoning approach, named ModCR. Compared to VLMs performing reasoning via cross modal semantic alignment, it regards the given textual abstract semantic and objective image information as the pre-context information and embeds them into the language model to perform context reasoning. Different from recent vision-aided language models used in natural language processing, ModCR incorporates the multi-view semantic alignment information between language and vision by introducing the learnable alignment prefix between image and text in the pretrained language model. This makes the language model well-suitable for such multi-modal reasoning scenario on joint textual and visual clues. We conduct extensive experiments on two corresponding data sets and experimental results show significantly improved performance (exact gain by 4.8% on PMR test set) compared to previous strong baselines. Code Link: \url{<a class="link-external link-https" href="https://github.com/YunxinLi/Multimodal-Context-Reasoning" rel="external noopener nofollow">this https URL</a>}.

The Classification of Multi-Modal Data with Hidden Conditional Random Field

Adaptive Video Segmentation Algorithm Using Hidden Conditional Random Fields

Adaptive Foreground and Shadow Segmentation Using Hidden Conditional Random Fields

Multi-view Heterogeneous Fusion and Embedding for Categorical Attributes on Mixed Data.

A Novel Hyperspectral Remote Sensing Images Classification Using Gaussian Processes with Conditional Random Fields

Conditional Random Field and Deep Feature Learning for Hyperspectral Image Classification

Muli-label Text Categorization with Hidden Components.

Part Detection, Description and Selection Based on Hidden Conditional Random Fields

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

A unified multimodal classification framework based on deep metric learning

Conditional Random Fields for Image Labeling

Joint multi-label multi-instance learning for image classification

Cross-Modal Learning to Rank Via Latent Joint Representation

Convolutional Neural Networks for Multimodal Remote Sensing Data Classification

A Multi-Modal Approach For Context-Aware Network Traffic Classification

Causal Conditional Hidden Markov Model for Multimodal Traffic Prediction

Multivariate time series classification based on fusion features

MRF-Based Multiple Classifier System for Hyperspectral Remote Sensing Image Classification.

Hierarchical Multi-label Text Classification with Horizontal and Vertical Category Correlations

Abnormal Detection based on Conditional Random Fields and Background Scene Partition

Hyperspectral Image Classification with Deep Metric Learning and Conditional Random Field