Abstract:How our brain encodes complex concepts has been a longstanding mystery in neuroscience. The answer to this problem can lead to new understandings about how the brain retrieves information in large-scale data with high efficiency and robustness. Neuroscience studies suggest the brain represents concepts in a locality-sensitive hashing (LSH) strategy, i.e., similar concepts will be represented by similar responses. This finding has inspired the design of similarity-based algorithms, especially in contrastive learning. Here, we hypothesize that the brain and large neural network models, both using similarity-based learning rules, could contain a similar semantic embedding space. To verify that, this paper proposes a functional Magnetic Resonance Imaging (fMRI) semantic learning network named BrainSem, aimed at seeking a joint semantic latent space that bridges the brain and a Contrastive Language-Image Pre-training (CLIP) model. Given that our perception is inherently cross-modal, we introduce a fuzzy (one-to-many) matching loss function to encourage the models to extract high-level semantic components from neural signals. Our results claimed that using only a small set of fMRI recordings for semantic space alignment, we could obtain shared embedding valid for unseen categories out of the training set, which provided potential evidence for the semantic representation similarity between the brain and large neural networks. In a zero-shot classification task, our BrainSem achieves an 11.6% improvement over the state-of-the-art.

Semantic Similarity Computing Model Based on Multi Model Fine-Grained Nonlinear Fusion

A Joint Model for Sentence Semantic Similarity Learning

Convolutional Network-Based Semantic Similarity Model of Sentences

Multi-Granularity Neural Sentence Model For Measuring Short Text Similarity

A Model for Chinese Sentence Similarity Computing

Improved Chinese Sentence Semantic Similarity Calculation Method Based on Multi-Feature Fusion

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Chinese Sentence Similarity Computational Model Based on Multi-Features Combination

Hybridization between Neural Computing and Nature-Inspired Algorithms for a Sentence Similarity Model Based on the Attention Mechanism

Sentences Similarity Model Based on Fusion of Semantic, Syntactic and Word Order Multi-Features

Sentence Similarity Computational Model Based on Information Content.

Chinese long text similarity calculation of semantic progressive fusion based on Bert

Fine-grained Semantic Textual Similarity Measurement Via a Feature Separation Network

MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similarity

Chinese Sentence Semantic Matching Based on Multi-Granularity Fusion Model

Sentence Pair Similarity Modeling Based on Weighted Interaction of Multi-semantic Embedding Matrix

Sentence Similarity Computation by Integrating Shallow and Deep Information

Model for Sentence Similarity Computing Based on Multi-Features Combination

Neural language model and semantic compositionality model in semantic simi-larity

A Novel Sentence Similarity Model with Word Embedding Based on Convolutional Neural Network.

Text Semantic Communication Systems with Sentence-Level Semantic Fidelity