Abstract:Classical value estimation reinforcement learning algorithms do not perform very well in dynamic environments. On the other hand, the reinforcement learning of animals is quite flexible: they can adapt to dynamic environments very quickly and deal with noisy inputs very effectively. One feature that may contribute to animals' good performance in dynamic environments is that they learn and perceive the time to reward. In this research, we attempt to learn and perceive the time to reward and explore situations where the learned time information can be used to improve the performance of the learning agent in dynamic environments. The type of dynamic environments that we are interested in is that type of switching environment which stays the same for a long time, then changes abruptly, and then holds for a long time before another change. The type of dynamics that we mainly focus on is the time to reward, though we also extend the ideas to learning and perceiving other criteria of optimality, e.g. the discounted return, so that they can still work even when the amount of reward may also change. Specifically, both the mean and variance of the time to reward are learned and then used to detect changes in the environment and to decide whether the agent should give up a suboptimal action. When a change in the environment is detected, the learning agent responds specifically to the change in order to recover quickly from it. When it is found that the current action is still worse than the optimal one, the agent gives up this time's exploration of the action and then remakes its decision in order to avoid longer than necessary exploration. The results of our experiments using two real-world problems show that they have effectively sped up learning, reduced the time taken to recover from environmental changes, and improved the performance of the agent after the learning converges in most of the test cases compared with classical value estimation reinforcement learning algorithms. In addition, we have successfully used spiking neurons to implement various phenomena of classical conditioning, the simplest form of animal reinforcement learning in dynamic environments, and also pointed out a possible implementation of instrumental conditioning and general reinforcement learning using similar models.

Expected reward value and reward prediction errors reinforce but also interfere with human time perception

Reward positivity affects temporal interval production in a continuous timing task

Dopamine transients follow a striatal gradient of reward time horizons

Dopamine transients encode reward prediction errors independent of learning rates

Mesolimbic dopamine encodes reward prediction errors independent of learning rates

Learning about reward identities and time

Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time

Reinforcement regulates timing variability in thalamus

Sub-second and multi-second dopamine dynamics underlie variability in human time perception

Reward timescale controls the rate of behavioral and dopaminergic learning

Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward

A general theory of intertemporal decision-making and the perception of time

Short-term Memory Traces for Action Bias in Human Reinforcement Learning

Influence of Recent Trial History on Interval Timing

The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Dopamine transients delivered in learning contexts do not act as model-free prediction errors

Representation and Timing in Theories of the Dopamine System

Reinforcement Learning with Time Perception

Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

Neural Repetition Suppression Modulates Time Perception: Evidence From Electrophysiology and Pupillometry