Abstract:The multimedia content analysis community has made significant effort to bridge the gap between low-level features and high-level semantics perceived by human cognitive systems such as real-world objects and concepts. In the two fields of multimedia analysis and brain imaging, both topics of low-level features and high level semantics are extensively studied. For instance, in the multimedia analysis field, many algorithms are available for multimedia feature extraction, and benchmark datasets are available such as the TRECVID. In the brain imaging field, brain regions that are responsible for vision, auditory perception, language, and working memory are well studied via functional magnetic resonance imaging (fMRI). This paper presents our initial effort in marrying these two fields in order to bridge the gaps between low-level features and high-level semantics via fMRI brain imaging. Our experimental paradigm is that we performed fMRI brain imaging when university student subjects watched the video clips selected from the TRECVID datasets. At current stage, we focus on the three concepts of sports, weather, and commercial-/advertisement specified in the TRECVID 2005. Meanwhile, the brain regions in vision, auditory, language, and working memory networks are quantitatively localized and mapped via task-based paradigm fMRI, and the fMRI responses in these regions are used to extract features as the representation of the brain's comprehension of semantics. Our computational framework aims to learn the most relevant low-level feature sets that best correlate the fMRI-derived semantics based on the training videos with fMRI scans, and then the learned models are applied to larger scale test datasets without fMRI scans for category classifications. Our result shows that: 1) there are meaningful couplings between brain's fMRI responses and video stimuli, suggesting the validity of linking semantics and low-level features via fMRI; 2) The computationally learned low-level feature sets from fMRI-derived semantic features can significantly improve the classification of video categories in comparison with that based on original low-level features.

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Alleviating the Semantic Gap for Generalized Fmri-to-image Reconstruction.

Exploring the Relationship Between Visual Information and Language Semantic Concept in the Human Brain

Finding Shared Decodable Concepts and their Negations in the Brain

Multimodal Learning via Exploring Deep Semantic Similarity.

Retrieving and reconstructing conceptually similar images from fMRI with latent diffusion models and a neuro-inspired brain decoding model

The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model

Bridging the Semantic Gap via Functional Brain Imaging

Decoupling Semantic Similarity from Spatial Alignment for Neural Networks

Multimodal Contrastive Learning for Brain-Machine Fusion: from Brain-in-the-loop Modeling to Brain-out-of-the-loop Application

Identifying Shared Decodable Concepts in the Human Brain Using Image-Language Foundation Models

Can computers understand words like humans do? Comparable semantic representation in neural and computer systems

Deep coupled registration and segmentation of multimodal whole-brain images

Joint fMRI Decoding and Encoding with Latent Embedding Alignment

Bridging Low-Level Features and High-Level Semantics Via Fmri Brain Imaging for Video Classification

Aligning Semantic in Brain and Language: A Curriculum Contrastive Method for Electroencephalography-to-Text Generation.

Semantic Neural Decoding via Cross-Modal Generation

Similarity-based fMRI-MEG fusion reveals hierarchical organisation within the brain's semantic system

Exploring Semantic Representation in Brain Activity Using Word Embeddings.

BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Models