Abstract:Classical value estimation reinforcement learning algorithms do not perform very well in dynamic environments. On the other hand, the reinforcement learning of animals is quite flexible: they can adapt to dynamic environments very quickly and deal with noisy inputs very effectively. One feature that may contribute to animals' good performance in dynamic environments is that they learn and perceive the time to reward. In this research, we attempt to learn and perceive the time to reward and explore situations where the learned time information can be used to improve the performance of the learning agent in dynamic environments. The type of dynamic environments that we are interested in is that type of switching environment which stays the same for a long time, then changes abruptly, and then holds for a long time before another change. The type of dynamics that we mainly focus on is the time to reward, though we also extend the ideas to learning and perceiving other criteria of optimality, e.g. the discounted return, so that they can still work even when the amount of reward may also change. Specifically, both the mean and variance of the time to reward are learned and then used to detect changes in the environment and to decide whether the agent should give up a suboptimal action. When a change in the environment is detected, the learning agent responds specifically to the change in order to recover quickly from it. When it is found that the current action is still worse than the optimal one, the agent gives up this time's exploration of the action and then remakes its decision in order to avoid longer than necessary exploration. The results of our experiments using two real-world problems show that they have effectively sped up learning, reduced the time taken to recover from environmental changes, and improved the performance of the agent after the learning converges in most of the test cases compared with classical value estimation reinforcement learning algorithms. In addition, we have successfully used spiking neurons to implement various phenomena of classical conditioning, the simplest form of animal reinforcement learning in dynamic environments, and also pointed out a possible implementation of instrumental conditioning and general reinforcement learning using similar models.

Time-scale invariant contingency yields one-shot reinforcement learning despite extremely long delays to reinforcement

Deep reinforcement learning with time-scale invariant memory

Reinforcement Learning with Time Perception

Rapid learning of temporal dependencies at multiple timescales

Short-term Memory Traces for Action Bias in Human Reinforcement Learning

Learning about reward identities and time

Modeling time perception in rats: Evidence for catastrophic interference in animal learning

The role of prospective contingency in the control of behavior and dopamine signals during associative learning

Continual Reinforcement Learning with Multi-Timescale Successor Features

Multi-timescale nexting in a reinforcement learning robot

Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning.

A neural network model for timing control with reinforcement

An Investigation of Time Reversal Symmetry in Reinforcement Learning

Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference

Provable Reinforcement Learning with a Short-Term Memory

A shared temporal window of integration across cognitive control and reinforcement learning paradigms: A correlational study

Demystifying the Recency Heuristic in Temporal-Difference Learning

Estimating scale-invariant future in continuous time

Reward timescale controls the rate of behavioral and dopaminergic learning

Temporal-Difference Learning Using Distributed Error Signals

One-shot learning for the long term: consolidation with an artificial hippocampal algorithm