Abstract:Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

Toward a data efficient neural actor-critic

Off-Policy Neural Fitted Actor-Critic

Neural fitted actor-critic.

Efficient Offline Reinforcement Learning: The Critic is Critical

Actor-Critic Alignment for Offline-to-Online Reinforcement Learning

The Actor-Dueling-Critic Method for Reinforcement Learning.

Graphical Object-Centric Actor-Critic

Recursive Least Squares Advantage Actor-Critic Algorithms

An efficient reinforcement learning algorithm for learning deterministic policies in continuous domains

Neural Network Compatible Off-Policy Natural Actor-Critic Algorithm

Reinforcement learning using a continuous time actor-critic framework with spiking neurons

Optimal Actor-Critic Policy With Optimized Training Datasets

Generalizing soft actor-critic algorithms to discrete action spaces

Generative Actor-Critic: An Off-policy Algorithm Using the Push-forward Model

Efficient Parallel Methods for Deep Reinforcement Learning

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

The Actor-Advisor: Policy Gradient With Off-Policy Advice

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning

A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor