Abstract:Cognitive maps are a proposed concept on how the brain efficiently organizes memories and retrieves context out of them. The entorhinal-hippocampal complex is heavily involved in episodic and relational memory processing, as well as spatial navigation and is thought to built cognitive maps via place and grid cells. To make use of the promising properties of cognitive maps, we set up a multi-modal neural network using successor representations which is able to model place cell dynamics and cognitive map representations. Here, we use multi-modal inputs consisting of images and word embeddings. The network learns the similarities between novel inputs and the training database and therefore the representation of the cognitive map successfully. Subsequently, the prediction of the network can be used to infer from one modality to another with over $90\%$ accuracy. The proposed method could therefore be a building block to improve current AI systems for better understanding of the environment and the different modalities in which objects appear. The association of specific modalities with certain encounters can therefore lead to context awareness in novel situations when similar encounters with less information occur and additional information can be inferred from the learned cognitive map. Cognitive maps, as represented by the entorhinal-hippocampal complex in the brain, organize and retrieve context from memories, suggesting that large language models (LLMs) like ChatGPT could harness similar architectures to function as a high-level processing center, akin to how the hippocampus operates within the cortex hierarchy. Finally, by utilizing multi-modal inputs, LLMs can potentially bridge the gap between different forms of data (like images and words), paving the way for context-awareness and grounding of abstract concepts through learned associations, addressing the grounding problem in AI.

Cross-modal associative memory by MultiSOM

Som Of Syntactic And Semantic Features Based On Chinese Sentences With Multi-Category Words

Word Learning Using a Self-Organizing Map

Simulation of Chinese characters learning with improved multi-SOM network

Cross-modal Retrieval Via Memory Network.

Learning Paired-associate Images with An Unsupervised Deep Learning Architecture

Deep Memory Network for Cross-Modal Retrieval

Self-Supervised Multimodal Domino: in Search of Biomarkers for Alzheimer's Disease

Multi-modal mutual topic reinforce modeling for cross-media retrieval

Cross-modality Representation Interactive Learning for Multimodal Sentiment Analysis

Multi-Modal Memory Enhancement Attention Network for Image-Text Matching

Multi-Modal Cognitive Maps based on Neural Networks trained on Successor Representations

Visual Image Classification Based on Auto-Associative Memory

Sparse Representation and Associative Learning in Multisensory Integration

The Framework and Memristive Circuit Design for Multisensory Mutual Associative Memory Networks

Self-organizing map analysis consistent with neuroimaging for chinese noun, verb and class-ambiguous word

A Multidirectional Associative Memory Based on Self-organizing Incremental Neural Network

Extracting the Brain-like Representation by an Improved Self-Organizing Map for Image Classification

Classification and Generation of real-world data with an Associative Memory Model

Multi-Modality Cross Attention Network for Image and Sentence Matching

Investigating Inner Properties of Multimodal Representation and Semantic Compositionality with Brain-based Componential Semantics