Abstract:Cognitive maps are a proposed concept on how the brain efficiently organizes memories and retrieves context out of them. The entorhinal-hippocampal complex is heavily involved in episodic and relational memory processing, as well as spatial navigation and is thought to built cognitive maps via place and grid cells. To make use of the promising properties of cognitive maps, we set up a multi-modal neural network using successor representations which is able to model place cell dynamics and cognitive map representations. Here, we use multi-modal inputs consisting of images and word embeddings. The network learns the similarities between novel inputs and the training database and therefore the representation of the cognitive map successfully. Subsequently, the prediction of the network can be used to infer from one modality to another with over $90\%$ accuracy. The proposed method could therefore be a building block to improve current AI systems for better understanding of the environment and the different modalities in which objects appear. The association of specific modalities with certain encounters can therefore lead to context awareness in novel situations when similar encounters with less information occur and additional information can be inferred from the learned cognitive map. Cognitive maps, as represented by the entorhinal-hippocampal complex in the brain, organize and retrieve context from memories, suggesting that large language models (LLMs) like ChatGPT could harness similar architectures to function as a high-level processing center, akin to how the hippocampus operates within the cortex hierarchy. Finally, by utilizing multi-modal inputs, LLMs can potentially bridge the gap between different forms of data (like images and words), paving the way for context-awareness and grounding of abstract concepts through learned associations, addressing the grounding problem in AI.

Multi-modal Cognitive Computing

Intelligence Methods of Multi-Modal Information Fusion in Human-Computer Interaction

Vision+X: A Survey on Multimodal Learning in the Light of Data

Multimodal Information Processing For Affective Computing

Cross-Modal Semantic Communications

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

AI-Enabled Cross-Modal Communications

Multimodal Machine Learning: A Survey and Taxonomy

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Multimodal Neuroimaging Computing: the Workflows, Methods, and Platforms.

Multi-Modal Cognitive Maps based on Neural Networks trained on Successor Representations

Multi-source brain computing with systematic fusion for smart health

Multi-modal Approach for Affective Computing

Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines

Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Cross-Modal Learning - The Learning Methodology Inspired by Human's Intelligence1

Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy

Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning

Emotion recognition based on brain-like multimodal hierarchical perception

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions