Deep Learning and Reward Design for Reinforcement Learning

Xiaoxiao Guo

Abstract:. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

Psychology,Computer Science

What problem does this paper attempt to address?

Reward Mechanism Design for Deep Reinforcement Learning-Based Microgrid Energy Management

Mingjie Hu,Baohui Han,Shilin Lv,Zhejing Bao,Lingxia Lu,Miao Yu

DOI: https://doi.org/10.1109/repe59476.2023.10512009

2023-01-01

Abstract:Deep Reinforcement Learning (DRL), with its data-driven nature and model-free advantage, has attracted great interest in the field of microgrid energy management system. The choice of reward mechanism plays a crucial role in the performance and effectiveness of DRL-based microgrid energy management. This paper aims to investigate the reward mechanism design by comparing the performances of DRL-based microgrid energy management under two different reward mechanisms, namely, cliff walking pattern and Leduc poker pattern. The reward mechanism incorporates auxiliary rewards alongside the primary reward to harmonize diverse objectives. Using a real microgrid dataset, the performance of DRL agents under different reward mechanisms are compared. The experimental results demonstrate that different reward mechanisms have a significant impact on the convergence speed and generalization ability of trained microgrid energy management policy.
Effective Reward Specification in Deep Reinforcement Learning

Julien Roy

2024-12-10

Abstract:In the last decade, Deep Reinforcement Learning has evolved into a powerful tool for complex sequential decision-making problems. It combines deep learning's proficiency in processing rich input signals with reinforcement learning's adaptability across diverse control tasks. At its core, an RL agent seeks to maximize its cumulative reward, enabling AI algorithms to uncover novel solutions previously unknown to experts. However, this focus on reward maximization also introduces a significant difficulty: improper reward specification can result in unexpected, misaligned agent behavior and inefficient learning. The complexity of accurately specifying the reward function is further amplified by the sequential nature of the task, the sparsity of learning signals, and the multifaceted aspects of the desired behavior. In this thesis, we survey the literature on effective reward specification strategies, identify core challenges relating to each of these approaches, and propose original contributions addressing the issue of sample efficiency and alignment in deep reinforcement learning. Reward specification represents one of the most challenging aspects of applying reinforcement learning in real-world domains. Our work underscores the absence of a universal solution to this complex and nuanced challenge; solving it requires selecting the most appropriate tools for the specific requirements of each unique application.

Machine Learning
Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards

Yixiang Wang,Yujing Hu,Feng Wu,Yingfeng Chen

DOI: https://doi.org/10.48550/arXiv.2207.14722

2022-07-29

Abstract:Reward design is a critical part of the application of reinforcement learning, the performance of which strongly depends on how well the reward signal frames the goal of the designer and how well the signal assesses progress in reaching that goal. In many cases, the extrinsic rewards provided by the environment (e.g., win or loss of a game) are very sparse and make it difficult to train agents directly. Researchers usually assist the learning of agents by adding some auxiliary rewards in practice. However, designing auxiliary rewards is often turned to a trial-and-error search for reward settings that produces acceptable results. In this paper, we propose to automatically generate goal-consistent intrinsic rewards for the agent to learn, by maximizing which the expected accumulative extrinsic rewards can be maximized. To this end, we introduce the concept of motivation which captures the underlying goal of maximizing certain rewards and propose the motivation based reward design method. The basic idea is to shape the intrinsic rewards by minimizing the distance between the intrinsic and extrinsic motivations. We conduct extensive experiments and show that our method performs better than the state-of-the-art methods in handling problems of delayed reward, exploration, and credit assignment.

Machine Learning,Artificial Intelligence
An Emotion-Based Approach to Reinforcement Learning Reward Design

Haixu Yu,Pei Yang

DOI: https://doi.org/10.1109/icnsc.2019.8743211

2019-01-01

Abstract:Reward is one of the crucial factors in reinforcement learning, which affects the improvement of control strategies. However, the role of reward design has received relatively little attention. In this paper, an emotion-based target reward function is proposed which requires the agent to possess the ability to reflect. In this approach, the learning process information of the agent is mapped to its internal changes in any episode. The difference in internal values of adjacent episodes induces the generation of the agent's emotions, which is a key way to assist the agent to internally measure preset external target reward. Our proposed approach is combined with traditional RL algorithms (i.e., Q-learning, Sarsa and Q(λ)-learning) to test its effectiveness. All experimental results show that emotion-based target reward can accelerate the learning process.
Deep Reinforcement Learning from Hierarchical Preference Design

Alexander Bukharin,Yixiao Li,Pengcheng He,Tuo Zhao

2024-06-10

Abstract:Reward design is a fundamental, yet challenging aspect of reinforcement learning (RL). Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals. This paper shows by exploiting certain structures, one can ease the reward design process. Specifically, we propose a hierarchical reward modeling framework -- HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning. Both scenarios allow us to design a hierarchical decision tree induced by the importance ranking of the feedback signals to compare RL trajectories. With such preference data, we can then train a reward model for policy learning. We apply HERON to several RL applications, and we find that our framework can not only train high performing agents on a variety of difficult tasks, but also provide additional benefits such as improved sample efficiency and robustness. Our code is available at \url{<a class="link-external link-https" href="https://github.com/abukharin3/HERON" rel="external noopener nofollow">this https URL</a>}.

Machine Learning,Artificial Intelligence
Reward Design with Language Models

Minae Kwon,Sang Michael Xie,Kalesha Bullard,Dorsa Sadigh

2023-02-28

Abstract:Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of the desired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning of training. During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal. The RL agent then uses this reward to update its behavior. We evaluate whether our approach can train agents aligned with user objectives in the Ultimatum Game, matrix games, and the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents trained with our framework are well-aligned with the user's objectives and outperform RL agents trained with reward functions learned via supervised learning

Machine Learning,Artificial Intelligence,Computation and Language
Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics

Jiayang Song,Zhehua Zhou,Jiawei Liu,Chunrong Fang,Zhan Shu,Lei Ma

DOI: https://doi.org/10.48550/arXiv.2309.06687

2023-10-03

Abstract:Although Deep Reinforcement Learning (DRL) has achieved notable success in numerous robotic applications, designing a high-performing reward function remains a challenging task that often requires substantial manual input. Recently, Large Language Models (LLMs) have been extensively adopted to address tasks demanding in-depth common-sense knowledge, such as reasoning and planning. Recognizing that reward function design is also inherently linked to such knowledge, LLM offers a promising potential in this context. Motivated by this, we propose in this work a novel LLM framework with a self-refinement mechanism for automated reward function design. The framework commences with the LLM formulating an initial reward function based on natural language inputs. Then, the performance of the reward function is assessed, and the results are presented back to the LLM for guiding its self-refinement process. We examine the performance of our proposed framework through a variety of continuous robotic control tasks across three diverse robotic systems. The results indicate that our LLM-designed reward functions are able to rival or even surpass manually designed reward functions, highlighting the efficacy and applicability of our approach.

Robotics,Artificial Intelligence
Towards Socially and Morally Aware RL agent: Reward Design With LLM

Zhaoyue Wang

2024-05-31

Abstract:When we design and deploy an Reinforcement Learning (RL) agent, reward functions motivates agents to achieve an objective. An incorrect or incomplete specification of the objective can result in behavior that does not align with human values - failing to adhere with social and moral norms that are ambiguous and context dependent, and cause undesired outcomes such as negative side effects and exploration that is unsafe. Previous work have manually defined reward functions to avoid negative side effects, use human oversight for safe exploration, or use foundation models as planning tools. This work studies the ability of leveraging Large Language Models (LLM)' understanding of morality and social norms on safe exploration augmented RL methods. This work evaluates language model's result against human feedbacks and demonstrates language model's capability as direct reward signals.

Artificial Intelligence
Tiered Reward: Designing Rewards for Specification and Fast Learning of Desired Behavior

Zhiyuan Zhou,Shreyas Sundara Raman,Henry Sowerby,Michael L. Littman

2024-08-02

Abstract:Reinforcement-learning agents seek to maximize a reward signal through environmental interactions. As humans, our job in the learning process is to design reward functions to express desired behavior and enable the agent to learn such behavior swiftly. However, designing good reward functions to induce the desired behavior is generally hard, let alone the question of which rewards make learning fast. In this work, we introduce a family of a reward structures we call Tiered Reward that addresses both of these questions. We consider the reward-design problem in tasks formulated as reaching desirable states and avoiding undesirable states. To start, we propose a strict partial ordering of the policy space to resolve trade-offs in behavior preference. We prefer policies that reach the good states faster and with higher probability while avoiding the bad states longer. Next, we introduce Tiered Reward, a class of environment-independent reward functions and show it is guaranteed to induce policies that are Pareto-optimal according to our preference relation. Finally, we demonstrate that Tiered Reward leads to fast learning with multiple tabular and deep reinforcement-learning algorithms.

Machine Learning,Artificial Intelligence
Deep Reinforcement Learning: An Overview

Yuxi Li

DOI: https://doi.org/10.48550/arXiv.1701.07274

2018-11-26

Abstract:We give an overview of recent exciting achievements of deep reinforcement learning (RL). We discuss six core elements, six important mechanisms, and twelve applications. We start with background of machine learning, deep learning and reinforcement learning. Next we discuss core RL elements, including value function, in particular, Deep Q-Network (DQN), policy, reward, model, planning, and exploration. After that, we discuss important mechanisms for RL, including attention and memory, unsupervised learning, transfer learning, multi-agent RL, hierarchical RL, and learning to learn. Then we discuss various applications of RL, including games, in particular, AlphaGo, robotics, natural language processing, including dialogue systems, machine translation, and text generation, computer vision, neural architecture design, business management, finance, healthcare, Industry 4.0, smart grid, intelligent transportation systems, and computer systems. We mention topics not reviewed yet, and list a collection of RL resources. After presenting a brief summary, we close with discussions. Please see Deep Reinforcement Learning, <a class="link-https" data-arxiv-id="1810.06339" href="https://arxiv.org/abs/1810.06339">arXiv:1810.06339</a>, for a significant update.

Machine Learning
Programmatic Reward Design by Example

Weichao Zhou,Wenchao Li

DOI: https://doi.org/10.48550/arXiv.2112.08438

2022-01-08

Abstract:Reward design is a fundamental problem in reinforcement learning (RL). A misspecified or poorly designed reward can result in low sample efficiency and undesired behaviors. In this paper, we propose the idea of programmatic reward design, i.e. using programs to specify the reward functions in RL environments. Programs allow human engineers to express sub-goals and complex task scenarios in a structured and interpretable way. The challenge of programmatic reward design, however, is that while humans can provide the high-level structures, properly setting the low-level details, such as the right amount of reward for a specific sub-task, remains difficult. A major contribution of this paper is a probabilistic framework that can infer the best candidate programmatic reward function from expert demonstrations. Inspired by recent generative-adversarial approaches, our framework searches for the most likely programmatic reward function under which the optimally generated trajectories cannot be differentiated from the demonstrated trajectories. Experimental results show that programmatic reward functionslearned using this framework can significantly outperform those learned using existing reward learning algo-rithms, and enable RL agents to achieve state-of-the-artperformance on highly complex tasks.

Machine Learning,Artificial Intelligence,Programming Languages
Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems

Yihao Feng,Shentao Yang,Shujian Zhang,Jianguo Zhang,Caiming Xiong,Mingyuan Zhou,Huan Wang

DOI: https://doi.org/10.48550/arxiv.2302.10342

2023-01-01

Abstract:When learning task-oriented dialogue (ToD) agents, reinforcement learning (RL) techniques can naturally be utilized to train dialogue strategies to achieve user-specific goals. Prior works mainly focus on adopting advanced RL techniques to train the ToD agents, while the design of the reward function is not well studied. This paper aims at answering the question of how to efficiently learn and leverage a reward function for training end-to-end (E2E) ToD agents. Specifically, we introduce two generalized objectives for reward-function learning, inspired by the classical learning-to-rank literature. Further, we utilize the learned reward function to guide the training of the E2E ToD agent. With the proposed techniques, we achieve competitive results on the E2E response-generation task on the Multiwoz 2.0 dataset. Source code and checkpoints are publicly released at https://github.com/Shentao-YANG/Fantastic_Reward_ICLR2023.
Secrets of RLHF in Large Language Models Part II: Reward Modeling

Binghai Wang,Rui Zheng,Lu Chen,Yan Liu,Shihan Dou,Caishuang Huang,Wei Shen,Senjie Jin,Enyu Zhou,Chenyu Shi,Songyang Gao,Nuo Xu,Yuhao Zhou,Xiaoran Fan,Zhiheng Xi,Jun Zhao,Xiao Wang,Tao Ji,Hang Yan,Lixing Shen,Zhan Chen,Tao Gui,Qi Zhang,Xipeng Qiu,Xuanjing Huang,Zuxuan Wu,Yu-Gang Jiang

2024-01-12

Abstract:Reinforcement Learning from Human Feedback (RLHF) has become a crucial technology for aligning language models with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive reinforcement learning optimization. While reward models are often considered central to achieving high performance, they face the following challenges in practical applications: (1) Incorrect and ambiguous preference pairs in the dataset may hinder the reward model from accurately capturing human intent. (2) Reward models trained on data from a specific distribution often struggle to generalize to examples outside that distribution and are not suitable for iterative RLHF training.

Artificial Intelligence
Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning

Shuyu Yin,Fei Wen,Peilin Liu,Tao Luo

2024-07-18

Abstract:In deep reinforcement learning applications, maximizing discounted reward is often employed instead of maximizing total reward to ensure the convergence and stability of algorithms, even though the performance metric for evaluating the policy remains the total reward. However, the optimal policies corresponding to these two objectives may not always be consistent. To address this issue, we analyzed the suboptimality of the policy obtained through maximizing discounted reward in relation to the policy that maximizes total reward and identified the influence of hyperparameters. Additionally, we proposed sufficient conditions for aligning the optimal policies of these two objectives under various settings. The primary contributions are as follows: We theoretically analyzed the factors influencing performance when using discounted reward as a proxy for total reward, thereby enhancing the theoretical understanding of this scenario. Furthermore, we developed methods to align the optimal policies of the two objectives in certain situations, which can improve the performance of reinforcement learning algorithms.

Machine Learning
Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications

Sinan Ibrahim,Mostafa Mostafa,Ali Jnadi,Pavel Osinenko

2024-07-22

Abstract:The aim of Reinforcement Learning (RL) in real-world applications is to create systems capable of making autonomous decisions by learning from their environment through trial and error. This paper emphasizes the importance of reward engineering and reward shaping in enhancing the efficiency and effectiveness of reinforcement learning algorithms. Reward engineering involves designing reward functions that accurately reflect the desired outcomes, while reward shaping provides additional feedback to guide the learning process, accelerating convergence to optimal policies. Despite significant advancements in reinforcement learning, several limitations persist. One key challenge is the sparse and delayed nature of rewards in many real-world scenarios, which can hinder learning progress. Additionally, the complexity of accurately modeling real-world environments and the computational demands of reinforcement learning algorithms remain substantial obstacles. On the other hand, recent advancements in deep learning and neural networks have significantly improved the capability of reinforcement learning systems to handle high-dimensional state and action spaces, enabling their application to complex tasks such as robotics, autonomous driving, and game playing. This paper provides a comprehensive review of the current state of reinforcement learning, focusing on the methodologies and techniques used in reward engineering and reward shaping. It critically analyzes the limitations and recent advancements in the field, offering insights into future research directions and potential applications in various domains.

Machine Learning,Systems and Control
A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning

Shengjie Sun,Runze Liu,Jiafei Lyu,Jing-Wen Yang,Liangpeng Zhang,Xiu Li

2024-10-19

Abstract:Large Language Models (LLMs) have shown significant potential in designing reward functions for Reinforcement Learning (RL) tasks. However, obtaining high-quality reward code often involves human intervention, numerous LLM queries, or repetitive RL training. To address these issues, we propose CARD, a LLM-driven Reward Design framework that iteratively generates and improves reward function code. Specifically, CARD includes a Coder that generates and verifies the code, while a Evaluator provides dynamic feedback to guide the Coder in improving the code, eliminating the need for human feedback. In addition to process feedback and trajectory feedback, we introduce Trajectory Preference Evaluation (TPE), which evaluates the current reward function based on trajectory preferences. If the code fails the TPE, the Evaluator provides preference feedback, avoiding RL training at every iteration and making the reward function better aligned with the task objective. Empirical results on Meta-World and ManiSkill2 demonstrate that our method achieves an effective balance between task performance and token efficiency, outperforming or matching the baselines across all tasks. On 10 out of 12 tasks, CARD shows better or comparable performance to policies trained with expert-designed rewards, and our method even surpasses the oracle on 3 tasks.

Machine Learning
Addressing Reward Engineering For Deep Reinforcement Learning On Multi-Stage Task

Bin Chen,Jianhua Su

DOI: https://doi.org/10.1007/978-3-030-36802-9_33

2019-01-01

Abstract:In the field of robotics, it is a challenge to deal with multi-stage tasks based on Deep reinforcement learning (Deep RL). Previous researches have shown manually shaping a reward function could easily result in sub-optimal performance, hence choosing a sparse reward is a natural and sensible decision in many cases. However, it is rare for the agent to explore a non-zero reward with the increase of the horizon under the sparse reward, which makes it difficult to learn an agent to deal with multi-stage task. In this paper, we aim to develop a Deep RL based policy through fully utilizing the demonstrations to address this problem. We use the learned policy to solve some difficult multi-stage tasks, such as picking-and-place, stacking blocks, and achieve good results. A video of our experiments can be found at: https://youtu.be/6BulNjqDg3I.
Text2Reward: Reward Shaping with Language Models for Reinforcement Learning

Tianbao Xie,Siheng Zhao,Chen Henry Wu,Yitao Liu,Qian Luo,Victor Zhong,Yanchao Yang,Tao Yu

2024-05-25

Abstract:Designing reward functions is a longstanding challenge in reinforcement learning (RL); it requires specialized knowledge or domain data, leading to high costs for development. To address this, we introduce Text2Reward, a data-free framework that automates the generation and shaping of dense reward functions based on large language models (LLMs). Given a goal described in natural language, Text2Reward generates shaped dense reward functions as an executable program grounded in a compact representation of the environment. Unlike inverse RL and recent work that uses LLMs to write sparse reward codes or unshaped dense rewards with a constant function across timesteps, Text2Reward produces interpretable, free-form dense reward codes that cover a wide range of tasks, utilize existing packages, and allow iterative refinement with human feedback. We evaluate Text2Reward on two robotic manipulation benchmarks (ManiSkill2, MetaWorld) and two locomotion environments of MuJoCo. On 13 of the 17 manipulation tasks, policies trained with generated reward codes achieve similar or better task success rates and convergence speed than expert-written reward codes. For locomotion tasks, our method learns six novel locomotion behaviors with a success rate exceeding 94%. Furthermore, we show that the policies trained in the simulator with our method can be deployed in the real world. Finally, Text2Reward further improves the policies by refining their reward functions with human feedback. Video results are available at <a class="link-external link-https" href="https://text-to-reward.github.io/" rel="external noopener nofollow">this https URL</a> .

Machine Learning,Artificial Intelligence,Computation and Language,Robotics
Self Punishment and Reward Backfill for Deep Q-Learning

Mohammad Reza Bonyadi,Rui Wang,Maryam Ziaei

DOI: https://doi.org/10.1109/TNNLS.2021.3140042

2022-01-02

Abstract:Reinforcement learning agents learn by encouraging behaviours which maximize their total reward, usually provided by the environment. In many environments, however, the reward is provided after a series of actions rather than each single action, leading the agent to experience ambiguity in terms of whether those actions are effective, an issue known as the credit assignment problem. In this paper, we propose two strategies inspired by behavioural psychology to enable the agent to intrinsically estimate more informative reward values for actions with no reward. The first strategy, called self-punishment (SP), discourages the agent from making mistakes that lead to undesirable terminal states. The second strategy, called the rewards backfill (RB), backpropagates the rewards between two rewarded actions. We prove that, under certain assumptions and regardless of the reinforcement learning algorithm used, these two strategies maintain the order of policies in the space of all possible policies in terms of their total reward, and, by extension, maintain the optimal policy. Hence, our proposed strategies integrate with any reinforcement learning algorithm that learns a value or action-value function through experience. We incorporated these two strategies into three popular deep reinforcement learning approaches and evaluated the results on thirty Atari games. After parameter tuning, our results indicate that the proposed strategies improve the tested methods in over 65 percent of tested games by up to over 25 times performance improvement.

Artificial Intelligence,Machine Learning
Exploring the design of reward functions in deep reinforcement learning-based vehicle velocity control algorithms

Yixu He,Yang Liu,Lan Yang,Xiaobo Qu

DOI: https://doi.org/10.1080/19427867.2024.2305018

2024-01-18

Transportation Letters

Abstract:The application of deep reinforcement learning (DRL) techniques in intelligent transportation systems garners significant attention. In this field, reward function design is a crucial factor for DRL performance. Current research predominantly relies on a trial-and-error approach for designing reward functions, lacking mathematical support and necessitating extensive empirical experimentation. Our research uses vehicle velocity control as a case study, build training and test sets, and develop a DRL framework for speed control. This framework examines both single-objective and multi-objective optimization in reward function designs. In single-objective optimization, we introduce "expected optimal velocity" as an optimization objective and analyze how different reward functions affect performance, providing a mathematical perspective on optimizing reward functions. In multi-objective optimization, we propose a reward function design paradigm and validate its effectiveness. Our findings offer a versatile framework and theoretical guidance for developing and optimizing reward functions in DRL, particularly for intelligent transportation systems.

transportation,transportation science & technology

Deep Learning and Reward Design for Reinforcement Learning

Reward Mechanism Design for Deep Reinforcement Learning-Based Microgrid Energy Management

Effective Reward Specification in Deep Reinforcement Learning

Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards

An Emotion-Based Approach to Reinforcement Learning Reward Design

Deep Reinforcement Learning from Hierarchical Preference Design

Reward Design with Language Models

Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics

Towards Socially and Morally Aware RL agent: Reward Design With LLM

Tiered Reward: Designing Rewards for Specification and Fast Learning of Desired Behavior

Deep Reinforcement Learning: An Overview

Programmatic Reward Design by Example

Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems

Secrets of RLHF in Large Language Models Part II: Reward Modeling

Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning

Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications

A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning

Addressing Reward Engineering For Deep Reinforcement Learning On Multi-Stage Task

Text2Reward: Reward Shaping with Language Models for Reinforcement Learning

Self Punishment and Reward Backfill for Deep Q-Learning

Exploring the design of reward functions in deep reinforcement learning-based vehicle velocity control algorithms