Abstract:Humans and non-human animals show great flexibility in spatial navigation, including the ability to return to specific locations based on as few as one single experience. To study spatial navigation in the laboratory, watermaze tasks, in which rats have to find a hidden platform in a pool of cloudy water surrounded by spatial cues, have long been used. Analogous tasks have been developed for human participants using virtual environments. Spatial learning in the watermaze is facilitated by the hippocampus. In particular, rapid, one-trial, allocentric place learning, as measured in the delayed-matching-to-place variant of the watermaze task, which requires rodents to learn repeatedly new locations in a familiar environment, is hippocampal dependent. In this article, we review some computational principles, embedded within a reinforcement learning framework, that utilise hippocampal spatial representations for navigation in watermaze tasks. We consider which key elements underlie their efficacy, and discuss their limitations in accounting for hippocampus-dependent navigation, both in terms of behavioural performance (i.e. how well do they reproduce behavioural measures of rapid place learning) and neurobiological realism (i.e. how well do they map to neurobiological substrates involved in rapid place learning). We discuss how an actor-critic architecture, enabling simultaneous assessment of the value of the current location and of the optimal direction to follow, can reproduce one-trial place learning performance as shown on watermaze and virtual delayed-matching-to-place tasks by rats and humans, respectively, if complemented with map-like place representations. The contribution of actor-critic mechanisms to delayed-matching-to-place performance is consistent with neurobiological findings implicating the striatum and hippocampo-striatal interaction in delayed-matching-to-place performance, given that the striatum has been associated with actor-critic mechanisms. Moreover, we illustrate that hierarchical computations embedded within an actor-critic architecture may help to account for aspects of flexible spatial navigation. The hierarchical reinforcement learning approach separates trajectory control via a temporal-difference error from goal selection via a goal prediction error and may account for flexible, trial-specific, navigation to familiar goal locations, as required in some arm-maze place memory tasks, although it does not capture one-trial learning of new goal locations, as observed in open field, including watermaze and virtual, delayed-matching-to-place tasks. Future models of one-shot learning of new goal locations, as observed on delayed-matching-to-place tasks, should incorporate hippocampal plasticity mechanisms that integrate new goal information with allocentric place representation, as such mechanisms are supported by substantial empirical evidence.

A Model of Place Field Reorganization During Reward Maximization

A Computational Theory of Learning Flexible Reward-Seeking Behavior with Place Cells

Neuro-Inspired Reinforcement Learning to Improve Trajectory Prediction in Reward-Guided Behavior

The Precision of Place Fields Governs Their Fate across Epochs of Experience

Spontaneous dynamics of hippocampal place fields in a model of combinatorial competition among stable inputs.

Reinforcement learning approaches to hippocampus-dependent flexible spatial navigation

A Brain-Inspired Model of Hippocampal Spatial Cognition Based on a Memory-Replay Mechanism

A computational model of learning flexible navigation in a maze by layout-conforming replay of place cells

Single field evolution rule governs the dynamics of representational drift in mouse hippocampal dorsal CA1 region.

A model of how hierarchical representations constructed in the hippocampus are used to navigate through space

Time Makes Space: Emergence of Place Fields in Networks Encoding Temporally Continuous Sensory Experiences

Integration of velocity-dependent spatio-temporal structure of place cell activation during navigation in a reservoir model of prefrontal cortex

Reward Cues in Space: Commonalities and Differences in Neural Coding by Hippocampal and Ventral Striatal Ensembles

Reward Coding of Hippocampal Neurons in Goal-directed Spatial Memory

Learning Allocentric Representations of Space for Navigation

Reward-based option competition in human dorsal stream and transition from stochastic exploration to exploitation in continuous space

Real-time sensory-motor integration of hippocampal place cell replay and prefrontal sequence learning in simulated and physical rat robots for novel path optimization

A model of hippocampal replay driven by experience and environmental structure facilitates spatial learning

Latent representations in hippocampal network model co-evolve with behavioral exploration of task structure

Predictive Coding of Reward in the Hippocampus

The cost of behavioral flexibility: reversal learning driven by a spiking neural network