Abstract:Classical value estimation reinforcement learning algorithms do not perform very well in dynamic environments. On the other hand, the reinforcement learning of animals is quite flexible: they can adapt to dynamic environments very quickly and deal with noisy inputs very effectively. One feature that may contribute to animals' good performance in dynamic environments is that they learn and perceive the time to reward. In this research, we attempt to learn and perceive the time to reward and explore situations where the learned time information can be used to improve the performance of the learning agent in dynamic environments. The type of dynamic environments that we are interested in is that type of switching environment which stays the same for a long time, then changes abruptly, and then holds for a long time before another change. The type of dynamics that we mainly focus on is the time to reward, though we also extend the ideas to learning and perceiving other criteria of optimality, e.g. the discounted return, so that they can still work even when the amount of reward may also change. Specifically, both the mean and variance of the time to reward are learned and then used to detect changes in the environment and to decide whether the agent should give up a suboptimal action. When a change in the environment is detected, the learning agent responds specifically to the change in order to recover quickly from it. When it is found that the current action is still worse than the optimal one, the agent gives up this time's exploration of the action and then remakes its decision in order to avoid longer than necessary exploration. The results of our experiments using two real-world problems show that they have effectively sped up learning, reduced the time taken to recover from environmental changes, and improved the performance of the agent after the learning converges in most of the test cases compared with classical value estimation reinforcement learning algorithms. In addition, we have successfully used spiking neurons to implement various phenomena of classical conditioning, the simplest form of animal reinforcement learning in dynamic environments, and also pointed out a possible implementation of instrumental conditioning and general reinforcement learning using similar models.

Time‐in‐action RL

Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning

Data Efficient Deep Reinforcement Learning with Action-Ranked Temporal Difference Learning

Reinforcement Learning with Time Perception

Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Driving

When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL

Generalized Reinforcement Learning: Experience Particles, Action Operator, Reinforcement Field, Memory Association, and Decision Concepts

Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control

Simplified Temporal Consistency Reinforcement Learning

Actor-Critic with variable time discretization via sustained actions

Hierarchical Reinforcement Learning for Kinematic Control Tasks with Parameterized Action Spaces

Temporal Difference Models: Model-Free Deep RL for Model-Based Control

Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off

Deep Reinforcement Learning With Macro-Actions

Reconstructing Actions To Explain Deep Reinforcement Learning

Safe reinforcement learning under temporal logic with reward design and quantum action selection

Temporal Shift Reinforcement Learning

An Investigation of Time Reversal Symmetry in Reinforcement Learning

Interval timing in deep reinforcement learning agents

Human-in-the-Loop Reinforcement Learning in Continuous-Action Space

Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning