Performance evaluation with temporal rewards

J. Voeten

DOI: https://doi.org/10.1016/S0166-5316(02)00105-0

2002-11-01

Abstract:

What problem does this paper attempt to address?

Learning Dense Reward with Temporal Variant Self-Supervision

Yuning Wu,Jieliang Luo,Hui Li

DOI: https://doi.org/10.48550/arXiv.2205.10431

IF: 5.414

2022-05-20

Machine Learning

Abstract:Rewards play an essential role in reinforcement learning. In contrast to rule-based game environments with well-defined reward functions, complex real-world robotic applications, such as contact-rich manipulation, lack explicit and informative descriptions that can directly be used as a reward. Previous effort has shown that it is possible to algorithmically extract dense rewards directly from multimodal observations. In this paper, we aim to extend this effort by proposing a more efficient and robust way of sampling and learning. In particular, our sampling approach utilizes temporal variance to simulate the fluctuating state and action distribution of a manipulation task. We then proposed a network architecture for self-supervised learning to better incorporate temporal information in latent representations. We tested our approach in two experimental setups, namely joint-assembly and door-opening. Preliminary results show that our approach is effective and efficient in learning dense rewards, and the learned rewards lead to faster convergence than baselines.
Self-Supervised Exploration via Temporal Inconsistency in Reinforcement Learning

Zijian Gao,Kele Xu,Yuanzhao Zhai,Dawei Feng,Bo Ding,XinJun Mao,Huaimin Wang

2023-06-27

Abstract:Under sparse extrinsic reward settings, reinforcement learning has remained challenging, despite surging interests in this field. Previous attempts suggest that intrinsic reward can alleviate the issue caused by sparsity. In this article, we present a novel intrinsic reward that is inspired by human learning, as humans evaluate curiosity by comparing current observations with historical knowledge. Our method involves training a self-supervised prediction model, saving snapshots of the model parameters, and using nuclear norm to evaluate the temporal inconsistency between the predictions of different snapshots as intrinsic rewards. We also propose a variational weighting mechanism to assign weight to different snapshots in an adaptive manner. Our experimental results on various benchmark environments demonstrate the efficacy of our method, which outperforms other intrinsic reward-based methods without additional training costs and with higher noise tolerance. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Machine Learning,Artificial Intelligence
Real Time Performance Evaluation for Dynamic Scheduling Algorithm

Lei Huang

1998-01-01

Journal of Applied Sciences

Abstract:On the basis of the dynamic scheduling algorithm of real time multitasking and the factors that be closely concerned with real time performance, a evaluation model is proposed. The model can point out that which tasks will produce time out fault and estimate the probability.
Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning

Gautham Vasan,Yan Wang,Fahim Shahriar,James Bergstra,Martin Jagersand,A. Rupam Mahmood

2024-07-09

Abstract:Many real-world robot learning problems, such as pick-and-place or arriving at a destination, can be seen as a problem of reaching a goal state as soon as possible. These problems, when formulated as episodic reinforcement learning tasks, can easily be specified to align well with our intended goal: -1 reward every time step with termination upon reaching the goal state, called minimum-time tasks. Despite this simplicity, such formulations are often overlooked in favor of dense rewards due to their perceived difficulty and lack of informativeness. Our studies contrast the two reward paradigms, revealing that the minimum-time task specification not only facilitates learning higher-quality policies but can also surpass dense-reward-based policies on their own performance metrics. Crucially, we also identify the goal-hit rate of the initial policy as a robust early indicator for learning success in such sparse feedback settings. Finally, using four distinct real-robotic platforms, we show that it is possible to learn pixel-based policies from scratch within two to three hours using constant negative rewards.

Robotics,Machine Learning
Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators

Mark Towers,Yali Du,Christopher Freeman,Timothy J. Norman

2024-08-15

Abstract:Future reward estimation is a core component of reinforcement learning agents; i.e., Q-value and state-value functions, predicting an agent's sum of future rewards. Their scalar output, however, obfuscates when or what individual future rewards an agent may expect to receive. We address this by modifying an agent's future reward estimator to predict their next N expected rewards, referred to as Temporal Reward Decomposition (TRD). This unlocks novel explanations of agent behaviour. Through TRD we can: estimate when an agent may expect to receive a reward, the value of the reward and the agent's confidence in receiving it; measure an input feature's temporal importance to the agent's action decisions; and predict the influence of different actions on future rewards. Furthermore, we show that DQN agents trained on Atari environments can be efficiently retrained to incorporate TRD with minimal impact on performance.

Artificial Intelligence,Machine Learning
Temporal Alignment for History Representation in Reinforcement Learning

Aleksandr Ermolov,Enver Sangineto,Nicu Sebe

DOI: https://doi.org/10.48550/arXiv.2204.03525

2022-04-07

Abstract:Environments in Reinforcement Learning are usually only partially observable. To address this problem, a possible solution is to provide the agent with information about the past. However, providing complete observations of numerous steps can be excessive. Inspired by human memory, we propose to represent history with only important changes in the environment and, in our approach, to obtain automatically this representation using self-supervision. Our method (TempAl) aligns temporally-close frames, revealing a general, slowly varying state of the environment. This procedure is based on contrastive loss, which pulls embeddings of nearby observations to each other while pushing away other samples from the batch. It can be interpreted as a metric that captures the temporal relations of observations. We propose to combine both common instantaneous and our history representation and we evaluate TempAl on all available Atari games from the Arcade Learning Environment. TempAl surpasses the instantaneous-only baseline in 35 environments out of 49. The source code of the method and of all the experiments is available at <a class="link-external link-https" href="https://github.com/htdt/tempal" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence
A Temporal Difference Reinforcement Learning Theory of Emotion: unifying emotion, cognition and adaptive behavior

Joost Broekens

DOI: https://doi.org/10.48550/arXiv.1807.08941

IF: 14.4

2018-07-24

Artificial Intelligence

Abstract:Emotions are intimately tied to motivation and the adaptation of behavior, and many animal species show evidence of emotions in their behavior. Therefore, emotions must be related to powerful mechanisms that aid survival, and, emotions must be evolutionary continuous phenomena. How and why did emotions evolve in nature, how do events get emotionally appraised, how do emotions relate to cognitive complexity, and, how do they impact behavior and learning? In this article I propose that all emotions are manifestations of reward processing, in particular Temporal Difference (TD) error assessment. Reinforcement Learning (RL) is a powerful computational model for the learning of goal oriented tasks by exploration and feedback. Evidence indicates that RL-like processes exist in many animal species. Key in the processing of feedback in RL is the notion of TD error, the assessment of how much better or worse a situation just became, compared to what was previously expected (or, the estimated gain or loss of utility - or well-being - resulting from new evidence). I propose a TDRL Theory of Emotion and discuss its ramifications for our understanding of emotions in humans, animals and machines, and present psychological, neurobiological and computational evidence in its support.
Expected reward value and reward prediction errors reinforce but also interfere with human time perception

Emily K. DiMarco,Ashley Ratcliffe Shipp,Kenneth T. Kishida

DOI: https://doi.org/10.1101/2024.04.17.589985

2024-04-21

Abstract:Time perception is often investigated in animal models and in humans using instrumental paradigms where reinforcement learning (RL) and associated dopaminergic processes have modulatory effects. For example, interval timing, which includes the judgment of relatively short intervals of time (ranging from milliseconds to minutes), has been shown to be modulated by manipulations of striatal dopamine. The ‘expected value of reward’ (EV) and ‘reward prediction errors’ (RPEs) are key variables described in RL-theory that explain dopaminergic signals during reward processing during instrumental learning. Notably, the underlying connection between RL-processes and time perception in humans is relatively underexplored. Herein, we investigated the impact of EV and RPEs on interval timing in humans. We tested the hypotheses that EV and RPEs modulate the experience of short time intervals. We demonstrate that expectations of monetary gains or losses increases the initial performance error for 1000ms intervals. Temporal learning over repeated trials is observed with accelerated learning of non-reinforced 1000ms intervals; however, RPEs – specifically about rewards and not punishments – appear to reinforce performance errors, which effectively interferes with the rate at which (reinforced) 1000ms intervals were learned. These effects were not significant for 3000ms and 5000ms intervals. Our results demonstrate that EV and RPEs influence human behavior about 1000ms time intervals. We discuss our results considering model-free ‘temporal difference RL-theory’, which suggests the hypothesis that interval timing may be mediated by dopaminergic signals that reinforce the learning and prediction of dynamic state-transitions which could be encoded without an explicit reference to ‘time’ intervals.

Neuroscience
Short-term Memory Traces for Action Bias in Human Reinforcement Learning

Rafal Bogacz,Samuel M. McClure,Jian Li,Jonathan D. Cohen,P. Read Montague

DOI: https://doi.org/10.1016/j.brainres.2007.03.057

IF: 3.61

2007-01-01

Brain Research

Abstract:Recent experimental and theoretical work on reinforcement learning has shed light on the neural bases of learning from rewards and punishments. One fundamental problem in reinforcement learning is the credit assignment problem, or how to properly assign credit to actions that lead to reward or punishment following a delay. Temporal difference learning solves this problem, but its efficiency can be significantly improved by the addition of eligibility traces (ET). In essence, ETs function as decaying memories of previous choices that are used to scale synaptic weight changes. It has been shown in theoretical studies that ETs spanning a number of actions may improve the performance of reinforcement learning. However, it remains an open question whether including ETs that persist over sequences of actions allows reinforcement learning models to better fit empirical data regarding the behaviors of humans and other animals. Here, we report an experiment in which human subjects performed a sequential economic decision game in which the long-term optimal strategy differed from the strategy that leads to the greatest short-term return. We demonstrate that human subjects' performance in the task is significantly affected by the time between choices in a surprising and seemingly counterintuitive way. However, this behavior is naturally explained by a temporal difference learning model which includes ETs persisting across actions. Furthermore, we review recent findings that suggest that short-term synaptic plasticity in dopamine neurons may provide a realistic biophysical mechanism for producing ETs that persist on a timescale consistent with behavioral observations.
Reward-Respecting Subtasks for Model-Based Reinforcement Learning

Richard S. Sutton,Marlos C. Machado,G. Zacharias Holland,David Szepesvari,Finbarr Timbers,Brian Tanner,Adam White

DOI: https://doi.org/10.1016/j.artint.2023.104001

2023-09-17

Abstract:To achieve the ambitious goals of artificial intelligence, reinforcement learning must include planning with a model of the world that is abstract in state and time. Deep learning has made progress with state abstraction, but temporal abstraction has rarely been used, despite extensively developed theory based on the options framework. One reason for this is that the space of possible options is immense, and the methods previously proposed for option discovery do not take into account how the option models will be used in planning. Options are typically discovered by posing subsidiary tasks, such as reaching a bottleneck state or maximizing the cumulative sum of a sensory signal other than reward. Each subtask is solved to produce an option, and then a model of the option is learned and made available to the planning process. In most previous work, the subtasks ignore the reward on the original problem, whereas we propose subtasks that use the original reward plus a bonus based on a feature of the state at the time the option terminates. We show that option models obtained from such reward-respecting subtasks are much more likely to be useful in planning than eigenoptions, shortest path options based on bottleneck states, or reward-respecting options generated by the option-critic. Reward respecting subtasks strongly constrain the space of options and thereby also provide a partial solution to the problem of option discovery. Finally, we show how values, policies, options, and models can all be learned online and off-policy using standard algorithms and general value functions.

Machine Learning,Artificial Intelligence
Temporal Shift Reinforcement Learning

Deepak George Thomas,Tichakorn Wongpiromsarn,Ali Jannesari

DOI: https://doi.org/10.48550/arXiv.2109.02145

2021-10-27

Abstract:The function approximators employed by traditional image-based Deep Reinforcement Learning (DRL) algorithms usually lack a temporal learning component and instead focus on learning the spatial component. We propose a technique, Temporal Shift Reinforcement Learning (TSRL), wherein both temporal, as well as spatial components are jointly learned. Moreover, TSRL does not require additional parameters to perform temporal learning. We show that TSRL outperforms the commonly used frame stacking heuristic on both of the Atari environments we test on while beating the SOTA for one of them. This investigation has implications in the robotics as well as sequential decision-making domains.

Machine Learning
Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards

Ahmad Ahmad,Mehdi Kermanshah,Kevin Leahy,Zachary Serlin,Ho Chit Siu,Makai Mann,Cristian-Ioan Vasile,Roberto Tron,Calin Belta

2024-12-04

Abstract:In this paper, we tackle the challenging problem of delayed rewards in reinforcement learning (RL). While Proximal Policy Optimization (PPO) has emerged as a leading Policy Gradient method, its performance can degrade under delayed rewards. We introduce two key enhancements to PPO: a hybrid policy architecture that combines an offline policy (trained on expert demonstrations) with an online PPO policy, and a reward shaping mechanism using Time Window Temporal Logic (TWTL). The hybrid architecture leverages offline data throughout training while maintaining PPO's theoretical guarantees. Building on the monotonic improvement framework of Trust Region Policy Optimization (TRPO), we prove that our approach ensures improvement over both the offline policy and previous iterations, with a bounded performance gap of $(2\varsigma\gamma\alpha^2)/(1-\gamma)^2$, where $\alpha$ is the mixing parameter, $\gamma$ is the discount factor, and $\varsigma$ bounds the expected advantage. Additionally, we prove that our TWTL-based reward shaping preserves the optimal policy of the original problem. TWTL enables formal translation of temporal objectives into immediate feedback signals that guide learning. We demonstrate the effectiveness of our approach through extensive experiments on an inverted pendulum and a lunar lander environments, showing improvements in both learning speed and final performance compared to standard PPO and offline-only approaches.

Machine Learning,Artificial Intelligence
Robot Policy Learning with Temporal Optimal Transport Reward

Yuwei Fu,Haichao Zhang,Di Wu,Wei Xu,Benoit Boulet

2024-10-29

Abstract:Reward specification is one of the most tricky problems in Reinforcement Learning, which usually requires tedious hand engineering in practice. One promising approach to tackle this challenge is to adopt existing expert video demonstrations for policy learning. Some recent work investigates how to learn robot policies from only a single/few expert video demonstrations. For example, reward labeling via Optimal Transport (OT) has been shown to be an effective strategy to generate a proxy reward by measuring the alignment between the robot trajectory and the expert demonstrations. However, previous work mostly overlooks that the OT reward is invariant to temporal order information, which could bring extra noise to the reward signal. To address this issue, in this paper, we introduce the Temporal Optimal Transport (TemporalOT) reward to incorporate temporal order information for learning a more accurate OT-based proxy reward. Extensive experiments on the Meta-world benchmark tasks validate the efficacy of the proposed method. Code is available at: <a class="link-external link-https" href="https://github.com/fuyw/TemporalOT" rel="external noopener nofollow">this https URL</a>

Artificial Intelligence,Robotics
A Hierarchical Reinforcement Learning Method for Persistent Time-Sensitive Tasks

Xiao Li,Calin Belta

DOI: https://doi.org/10.48550/arXiv.1606.06355

2016-06-21

Abstract:Reinforcement learning has been applied to many interesting problems such as the famous TD-gammon and the inverted helicopter flight. However, little effort has been put into developing methods to learn policies for complex persistent tasks and tasks that are time-sensitive. In this paper, we take a step towards solving this problem by using signal temporal logic (STL) as task specification, and taking advantage of the temporal abstraction feature that the options framework provide. We show via simulation that a relatively easy to implement algorithm that combines STL and options can learn a satisfactory policy with a small number of training cases

Artificial Intelligence
Effects of sparse rewards of different magnitudes in the speed of learning of model-based actor critic methods

Juan Vargas,Lazar Andjelic,Amir Barati Farimani

DOI: https://doi.org/10.48550/arXiv.2001.06725

2020-01-19

Abstract:Actor critic methods with sparse rewards in model-based deep reinforcement learning typically require a deterministic binary reward function that reflects only two possible outcomes: if, for each step, the goal has been achieved or not. Our hypothesis is that we can influence an agent to learn faster by applying an external environmental pressure during training, which adversely impacts its ability to get higher rewards. As such, we deviate from the classical paradigm of sparse rewards and add a uniformly sampled reward value to the baseline reward to show that (1) sample efficiency of the training process can be correlated to the adversity experienced during training, (2) it is possible to achieve higher performance in less time and with less resources, (3) we can reduce the performance variability experienced seed over seed, (4) there is a maximum point after which more pressure will not generate better results, and (5) that random positive incentives have an adverse effect when using a negative reward strategy, making an agent under those conditions learn poorly and more slowly. These results have been shown to be valid for Deep Deterministic Policy Gradients using Hindsight Experience Replay in a well known Mujoco environment, but we argue that they could be generalized to other methods and environments as well.

Machine Learning,Robotics
A Policy Search Method For Temporal Logic Specified Reinforcement Learning Tasks

Xiao Li,Yao Ma,Calin Belta

DOI: https://doi.org/10.48550/arXiv.1709.09611

2017-09-28

Abstract:Reward engineering is an important aspect of reinforcement learning. Whether or not the user's intentions can be correctly encapsulated in the reward function can significantly impact the learning outcome. Current methods rely on manually crafted reward functions that often require parameter tuning to obtain the desired behavior. This operation can be expensive when exploration requires systems to interact with the physical world. In this paper, we explore the use of temporal logic (TL) to specify tasks in reinforcement learning. TL formula can be translated to a real-valued function that measures its level of satisfaction against a trajectory. We take advantage of this function and propose temporal logic policy search (TLPS), a model-free learning technique that finds a policy that satisfies the TL specification. A set of simulated experiments are conducted to evaluate the proposed approach.

Artificial Intelligence
Defining Admissible Rewards for High Confidence Policy Evaluation

Niranjani Prasad,Barbara E Engelhardt,Finale Doshi-Velez

DOI: https://doi.org/10.48550/arXiv.1905.13167

2019-05-31

Abstract:A key impediment to reinforcement learning (RL) in real applications with limited, batch data is defining a reward function that reflects what we implicitly know about reasonable behaviour for a task and allows for robust off-policy evaluation. In this work, we develop a method to identify an admissible set of reward functions for policies that (a) do not diverge too far from past behaviour, and (b) can be evaluated with high confidence, given only a collection of past trajectories. Together, these ensure that we propose policies that we trust to be implemented in high-risk settings. We demonstrate our approach to reward design on synthetic domains as well as in a critical care context, for a reward that consolidates clinical objectives to learn a policy for weaning patients from mechanical ventilation.

Machine Learning
Off-Policy Reinforcement Learning with Delayed Rewards

Beining Han,Zhizhou Ren,Zuofan Wu,Yuan Zhou,Jian Peng

2022-01-01

Abstract:We study deep reinforcement learning (RL) algorithms with delayed rewards. In many real-world tasks, instant rewards are often not readily accessible or even defined immediately after the agent performs actions. In this work, we first formally define the environment with delayed rewards and discuss the challenges raised due to the non-Markovian nature of such environments. Then, we introduce a general off-policy RL framework with a new Q-function formulation that can handle the delayed rewards with theoretical convergence guarantees. For practical tasks with high dimensional state spaces, we further introduce the HC-decomposition rule of the Q-function in our framework which naturally leads to an approximation scheme that helps boost the training efficiency and stability. We finally conduct extensive experiments to demonstrate the superior performance of our algorithms over the existing work and their variants.
Learning about reward identities and time

Andrew R Delamater,Daniel B Siegel,Norman C Tu

DOI: https://doi.org/10.1016/j.beproc.2023.104859

Abstract:We discuss three empirical findings that we think any theory attempting to integrate interval timing with associative learning concepts will need to address. These empirical phenomena all come from studies that combine peak timing procedures with reinforcer devaluation or conditional discrimination tasks commonly employed, respectively, in interval timing or associative learning research traditions. The three phenomena we discuss include: (1) the observation that disruptions in reward identity encoding have little to no impact on the encoding of reward time in the peak procedure (Delamateret al., 2018), (2) the findings that organisms tend to average their time estimates when presented with a stimulus compound consisting of separately learned stimuli indicating short or long reward times but that such temporal averaging, itself, is sensitive to post-conditioning selective reward devaluation, and (3) that rats can learn a temporal patterning task in which two stimuli presented independently indicate one time to reward availability while their compound indicates another. We review our prior results and present new findings illustrating these three phenomena and we discuss the special challenges they pose for cascade theories of timing, for multiple-oscillator models, and for any approach that attempts to integrate interval timing and associative models. We close by illustrating some ways in which multi-layer connectionist network models might begin to address some of our key findings. We believe this will require an approach that includes separate mechanisms that code for reward identity and time, but that does so in a way that permits for integration between the two systems.
Temporal-adaptive Hierarchical Reinforcement Learning

Zhou Wen-Ji,Yu Yang

2020-01-01

Abstract: Hierarchical reinforcement learning (HRL) helps address large-scale and sparse reward issues in reinforcement learning. In HRL, the policy model has an inner representation structured in levels. With this structure, the reinforcement learning task is expected to be decomposed into corresponding levels with sub-tasks, and thus the learning can be more efficient. In HRL, although it is intuitive that a high-level policy only needs to make macro decisions in a low frequency, the exact frequency is hard to be simply determined. Previous HRL approaches often employed a fixed-time skip strategy or learn a terminal condition without taking account of the context, which, however, not only requires manual adjustments but also sacrifices some decision granularity. In this paper, we propose the \emph{temporal-adaptive hierarchical policy learning} (TEMPLE) structure, which uses a temporal gate to adaptively control the high-level policy decision frequency. We train the TEMPLE structure with PPO and test its performance in a range of environments including 2-D rooms, Mujoco tasks, and Atari games. The results show that the TEMPLE structure can lead to improved performance in these environments with a sequential adaptive high-level control.

Performance evaluation with temporal rewards

Learning Dense Reward with Temporal Variant Self-Supervision

Self-Supervised Exploration via Temporal Inconsistency in Reinforcement Learning

Real Time Performance Evaluation for Dynamic Scheduling Algorithm

Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning

Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators

Temporal Alignment for History Representation in Reinforcement Learning

A Temporal Difference Reinforcement Learning Theory of Emotion: unifying emotion, cognition and adaptive behavior

Expected reward value and reward prediction errors reinforce but also interfere with human time perception

Short-term Memory Traces for Action Bias in Human Reinforcement Learning

Reward-Respecting Subtasks for Model-Based Reinforcement Learning

Temporal Shift Reinforcement Learning

Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards

Robot Policy Learning with Temporal Optimal Transport Reward

A Hierarchical Reinforcement Learning Method for Persistent Time-Sensitive Tasks

Effects of sparse rewards of different magnitudes in the speed of learning of model-based actor critic methods

A Policy Search Method For Temporal Logic Specified Reinforcement Learning Tasks

Defining Admissible Rewards for High Confidence Policy Evaluation

Off-Policy Reinforcement Learning with Delayed Rewards

Learning about reward identities and time

Temporal-adaptive Hierarchical Reinforcement Learning