Abstract:Decoding human visual neural representations is a challenging task with great scientific significance in revealing vision-processing mechanisms and developing brain-like intelligent machines. Most existing methods are difficult to generalize to novel categories that have no corresponding neural data for training. The two main reasons are 1) the under-exploitation of the multimodal semantic knowledge underlying the neural data and 2) the small number of paired (stimuli-responses) training data. To overcome these limitations, this paper presents a generic neural decoding method called BraVL that uses multimodal learning of brain-visual-linguistic features. We focus on modeling the relationships between brain, visual and linguistic features via multimodal deep generative models. Specifically, we leverage the mixture-of-product-of-experts formulation to infer a latent code that enables a coherent joint generation of all three modalities. To learn a more consistent joint representation and improve the data efficiency in the case of limited brain activity data, we exploit both intra- and inter-modality mutual information maximization regularization terms. In particular, our BraVL model can be trained under various semi-supervised scenarios to incorporate the visual and textual features obtained from the extra categories. Finally, we construct three trimodal matching datasets, and the extensive experiments lead to some interesting conclusions and cognitive insights: 1) decoding novel visual categories from human brain activity is practically possible with good accuracy; 2) decoding models using the combination of visual and linguistic features perform much better than those using either of them alone; 3) visual perception may be accompanied by linguistic influences to represent the semantics of visual stimuli.

A Visually Enhanced Neural Encoder for Synset Induction

Visual Oriented Encoder: Integrating Multimodal and Multi-Scale Contexts for Video Captioning

Enhancing Sentence Representation with Visually-supervised Multimodal Pre-training

A Multimodal Visual Encoding Model Aided by Introducing Verbal Semantic Information

GiVE: Guiding Visual Encoder to Perceive Overlooked Information

Multimodal Tag Synset Induction

Attention-enhanced Joint Learning Network for Micro-Video Venue Classification

End-to-End Visual Grounding Framework for Multimodal NER in Social Media Posts

Syncretic Space Learning Network for NIR-VIS Face Recognition

Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training

Multimodal Learning with Triplet Ranking Loss for Visual Semantic Embedding Learning

GaborNet Visual Encoding: A Lightweight Region-Based Visual Encoding Model With Good Expressiveness and Biological Interpretability

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

Enhance audio-visual segmentation with hierarchical encoder and audio guidance

Multimodality-guided Visual-Caption Semantic Enhancement

Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval

Mutual Complementarity: Multi-Modal Enhancement Semantic Learning for Micro-Video Scene Recognition

Multimodal Autoregressive Pre-training of Large Vision Encoders

ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphs

MVPN: Multi-granularity visual prompt-guided fusion network for multimodal named entity recognition

Unified Visual-Semantic Embeddings: Bridging Vision and Language with Structured Meaning Representations