Abstract:Cognitive maps are a proposed concept on how the brain efficiently organizes memories and retrieves context out of them. The entorhinal-hippocampal complex is heavily involved in episodic and relational memory processing, as well as spatial navigation and is thought to built cognitive maps via place and grid cells. To make use of the promising properties of cognitive maps, we set up a multi-modal neural network using successor representations which is able to model place cell dynamics and cognitive map representations. Here, we use multi-modal inputs consisting of images and word embeddings. The network learns the similarities between novel inputs and the training database and therefore the representation of the cognitive map successfully. Subsequently, the prediction of the network can be used to infer from one modality to another with over $90\%$ accuracy. The proposed method could therefore be a building block to improve current AI systems for better understanding of the environment and the different modalities in which objects appear. The association of specific modalities with certain encounters can therefore lead to context awareness in novel situations when similar encounters with less information occur and additional information can be inferred from the learned cognitive map. Cognitive maps, as represented by the entorhinal-hippocampal complex in the brain, organize and retrieve context from memories, suggesting that large language models (LLMs) like ChatGPT could harness similar architectures to function as a high-level processing center, akin to how the hippocampus operates within the cortex hierarchy. Finally, by utilizing multi-modal inputs, LLMs can potentially bridge the gap between different forms of data (like images and words), paving the way for context-awareness and grounding of abstract concepts through learned associations, addressing the grounding problem in AI.

Neural Map: Structured Memory for Deep Reinforcement Learning

Spiking Reinforcement Learning with Memory Ability for Mapless Navigation

Cognitive Navigation by Neuro-Inspired Localization, Mapping, and Episodic Memory

Neural SLAM: Learning to Explore with External Memory

Cognitive Memory and Mapping in a Brain-Like System for Robotic Navigation

Multi-Object Navigation with dynamically learned neural implicit representations

Map-based Experience Replay: A Memory-Efficient Solution to Catastrophic Forgetting in Reinforcement Learning

Learning a World Model With Multitimescale Memory Augmentation

A Novel Neural Multi-Store Memory Network for Autonomous Visual Navigation in Unknown Environment

Vision Enhanced Neuro-Cognitive Structure for Robotic Spatial Cognition

Multigrid Neural Memory

A Navigation Cognitive System Driven by Hierarchical Spiking Neural Network.

Scalable Spatial Memory for Scene Rendering and Navigation

Neural network based successor representations to form cognitive maps of space and language

Sparse Graphical Memory for Robust Planning

Relational recurrent neural networks

Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks

Multi-Modal Cognitive Maps based on Neural Networks trained on Successor Representations

A Novel Mapping Strategy Based on Neocortex Model: Pre-liminary Results by Hierarchical Temporal Memory

Sample Efficient Reinforcement Learning Using Graph-Based Memory Reconstruction.

Structured Scene Memory for Vision-Language Navigation