Abstract:How our brain encodes complex concepts has been a longstanding mystery in neuroscience. The answer to this problem can lead to new understandings about how the brain retrieves information in large-scale data with high efficiency and robustness. Neuroscience studies suggest the brain represents concepts in a locality-sensitive hashing (LSH) strategy, i.e., similar concepts will be represented by similar responses. This finding has inspired the design of similarity-based algorithms, especially in contrastive learning. Here, we hypothesize that the brain and large neural network models, both using similarity-based learning rules, could contain a similar semantic embedding space. To verify that, this paper proposes a functional Magnetic Resonance Imaging (fMRI) semantic learning network named BrainSem, aimed at seeking a joint semantic latent space that bridges the brain and a Contrastive Language-Image Pre-training (CLIP) model. Given that our perception is inherently cross-modal, we introduce a fuzzy (one-to-many) matching loss function to encourage the models to extract high-level semantic components from neural signals. Our results claimed that using only a small set of fMRI recordings for semantic space alignment, we could obtain shared embedding valid for unseen categories out of the training set, which provided potential evidence for the semantic representation similarity between the brain and large neural networks. In a zero-shot classification task, our BrainSem achieves an 11.6% improvement over the state-of-the-art.

A Joint Model for Sentence Semantic Similarity Learning

Semantic Similarity Computing Model Based on Multi Model Fine-Grained Nonlinear Fusion

Convolutional Network-Based Semantic Similarity Model of Sentences

Sentence Similarity Computation by Integrating Shallow and Deep Information

Sentence Similarity Based on Semantic Vector Model.

Improved Chinese Sentence Semantic Similarity Calculation Method Based on Multi-Feature Fusion

A Novel Sentence Similarity Model with Word Embedding Based on Convolutional Neural Network.

Ercnn: Enhanced Recurrent Convolutional Neural Networks For Learning Sentence Similarity

A Model for Chinese Sentence Similarity Computing

Sentence Pair Similarity Modeling Based on Weighted Interaction of Multi-semantic Embedding Matrix

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Sentence Similarity Computational Model Based on Information Content.

An Efficient Framework for Sentence Similarity Modeling.

Sentence Similarity Based on Contexts

Text Semantic Communication Systems with Sentence-Level Semantic Fidelity

Sentence Similarity Model Based on Semantics and Syntax

Hybridization between Neural Computing and Nature-Inspired Algorithms for a Sentence Similarity Model Based on the Attention Mechanism

Research on Sentence Semantic Similarity Calculation Based on Word2vec

Siamese Recurrent Architectures for Learning Sentence Similarity

Learning Sentences Similarity by Multi-Head Attention

Chinese Sentence Similarity Computational Model Based on Multi-Features Combination