A model-based reinforcement learning method based on conditional generative adversarial networks

Tingting Zhao,Ying Wang,Guixi Li,Le Kong,Yarui Chen,Yuan Wang,Ning Xie,Jucheng Yang

DOI: https://doi.org/10.1016/j.patrec.2021.08.019

IF: 4.757

2021-12-01

Pattern Recognition Letters

Abstract:Generative adversarial networks, Model-based reinforcement learning, Policy learning, Transition model learning

computer science, artificial intelligence

What problem does this paper attempt to address?

Model Gradient: Unified Model and Policy Learning in Model-Based Reinforcement Learning

Chengxing Jia,Fuxiang Zhang,Tian Xu,Jing-Cheng Pang,Zongzhang Zhang,Yang Yu

DOI: https://doi.org/10.1007/s11704-023-3150-5

IF: 2.6688

2023-01-01

Frontiers of Computer Science

Abstract:Model-based reinforcement learning is a promising direction to improve the sample efficiency of reinforcement learning with learning a model of the environment. Previous model learning methods aim at fitting the transition data, and commonly employ a supervised learning approach to minimize the distance between the predicted state and the real state. The supervised model learning methods, however, diverge from the ultimate goal of model learning, i.e., optimizing the learned-in-the-model policy. In this work, we investigate how model learning and policy learning can share the same objective of maximizing the expected return in the real environment. We find model learning towards this objective can result in a target of enhancing the similarity between the gradient on generated data and the gradient on the real data. We thus derive the gradient of the model from this target and propose the Model Gradient algorithm (MG) to integrate this novel model learning approach with policy-gradient-based policy optimization. We conduct experiments on multiple locomotion control tasks and find that MG can not only achieve high sample efficiency but also lead to better convergence performance compared to traditional model-based reinforcement learning approaches.
Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective

Jinouwen Zhang,Rongkun Xue,Yazhe Niu,Yun Chen,Jing Yang,Hongsheng Li,Yu Liu

2024-12-02

Abstract:Generative models, particularly diffusion models, have achieved remarkable success in density estimation for multimodal data, drawing significant interest from the reinforcement learning (RL) community, especially in policy modeling in continuous action spaces. However, existing works exhibit significant variations in training schemes and RL optimization objectives, and some methods are only applicable to diffusion models. In this study, we compare and analyze various generative policy training and deployment techniques, identifying and validating effective designs for generative policy algorithms. Specifically, we revisit existing training objectives and classify them into two categories, each linked to a simpler approach. The first approach, Generative Model Policy Optimization (GMPO), employs a native advantage-weighted regression formulation as the training objective, which is significantly simpler than previous methods. The second approach, Generative Model Policy Gradient (GMPG), offers a numerically stable implementation of the native policy gradient method. We introduce a standardized experimental framework named GenerativeRL. Our experiments demonstrate that the proposed methods achieve state-of-the-art performance on various offline-RL datasets, offering a unified and practical guideline for training and deploying generative policies.

Machine Learning,Artificial Intelligence
Model-Based Reinforcement Learning via Meta-Policy Optimization

Ignasi Clavera,Jonas Rothfuss,John Schulman,Yasuhiro Fujita,Tamim Asfour,Pieter Abbeel

DOI: https://doi.org/10.48550/arXiv.1809.05214

2018-09-14

Abstract:Model-based reinforcement learning approaches carry the promise of being data efficient. However, due to challenges in learning dynamics models that sufficiently match the real-world dynamics, they struggle to achieve the same asymptotic performance as model-free methods. We propose Model-Based Meta-Policy-Optimization (MB-MPO), an approach that foregoes the strong reliance on accurate learned dynamics models. Using an ensemble of learned dynamic models, MB-MPO meta-learns a policy that can quickly adapt to any model in the ensemble with one policy gradient step. This steers the meta-policy towards internalizing consistent dynamics predictions among the ensemble while shifting the burden of behaving optimally w.r.t. the model discrepancies towards the adaptation step. Our experiments show that MB-MPO is more robust to model imperfections than previous model-based approaches. Finally, we demonstrate that our approach is able to match the asymptotic performance of model-free methods while requiring significantly less experience.

Machine Learning,Artificial Intelligence
Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

Chengyang Ying,Jun Zhu,Hang Su,Cheng Lu,Huayu Chen

DOI: https://doi.org/10.48550/arXiv.2209.14548

2022-09-29

Abstract:In offline reinforcement learning, weighted regression is a common method to ensure the learned policy stays close to the behavior policy and to prevent selecting out-of-sample actions. In this work, we show that due to the limited distributional expressivity of policy models, previous methods might still select unseen actions during training, which deviates from their initial motivation. To address this problem, we adopt a generative approach by decoupling the learned policy into two parts: an expressive generative behavior model and an action evaluation model. The key insight is that such decoupling avoids learning an explicitly parameterized policy model with a closed-form expression. Directly learning the behavior policy allows us to leverage existing advances in generative modeling, such as diffusion-based methods, to model diverse behaviors. As for action evaluation, we combine our method with an in-sample planning technique to further avoid selecting out-of-sample actions and increase computational efficiency. Experimental results on D4RL datasets show that our proposed method achieves competitive or superior performance compared with state-of-the-art offline RL methods, especially in complex tasks such as AntMaze. We also empirically demonstrate that our method can successfully learn from a heterogeneous dataset containing multiple distinctive but similarly successful strategies, whereas previous unimodal policies fail.

Computer Science
Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference

Xiaocong Chen,Lina Yao,Xianzhi Wang,Aixin Sun,Wenjie Zhang,Quan Z. Sheng

DOI: https://doi.org/10.48550/arXiv.2105.00822

2021-05-05

Abstract:Recent advances in reinforcement learning have inspired increasing interest in learning user modeling adaptively through dynamic interactions, e.g., in reinforcement learning based recommender systems. Reward function is crucial for most of reinforcement learning applications as it can provide the guideline about the optimization. However, current reinforcement-learning-based methods rely on manually-defined reward functions, which cannot adapt to dynamic and noisy environments. Besides, they generally use task-specific reward functions that sacrifice generalization ability. We propose a generative inverse reinforcement learning for user behavioral preference modelling, to address the above issues. Instead of using predefined reward functions, our model can automatically learn the rewards from user's actions based on discriminative actor-critic network and Wasserstein GAN. Our model provides a general way of characterizing and explaining underlying behavioral tendencies, and our experiments show our method outperforms state-of-the-art methods in a variety of scenarios, namely traffic signal control, online recommender systems, and scanpath prediction.

Machine Learning,Artificial Intelligence,Information Retrieval
Intelligent Model Learning Based on Variance for Bayesian Reinforcement Learning

Shuhua You,Quan Liu,Zongzhang Zhang,Hui Wang,Xiaofang Zhang

DOI: https://doi.org/10.1109/ictai.2015.37

2015-01-01

Abstract:We consider a modular method to reinforcement learning that represents uncertainty of model parameters by maintaining probability distributions over them. The algorithm we call MBDP (model-based Bayesian dynamic programming) can be decomposed into two parallel types of inference: model learning and policy learning. During learning a model, we update posterior distributions of a model over observations after taking an action in each state. During learning a policy, we solve MDPs by dynamic programming with greedy approximation to make an agent choose behaviors which maximize return under the estimated model. Furthermore, we propose a principled method which utilizes the variance of Dirichlet distributions for determining when to learn and relearn the model. We demonstrate that MBDP can find near optimal policies with high probability by sufficient model learning and experimental results show that MBDP performs better compared with current state-of-the-art methods in reinforcement learning.
VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control

Xingxing Liang,Qi Wang,Yanghe Feng,Zhong Liu,Jincai Huang

DOI: https://doi.org/10.48550/arXiv.1812.09968

IF: 5.414

2018-12-24

Machine Learning

Abstract:Recent breakthroughs in Go play and strategic games have witnessed the great potential of reinforcement learning in intelligently scheduling in uncertain environment, but some bottlenecks are also encountered when we generalize this paradigm to universal complex tasks. Among them, the low efficiency of data utilization in model-free reinforcement algorithms is of great concern. In contrast, the model-based reinforcement learning algorithms can reveal underlying dynamics in learning environments and seldom suffer the data utilization problem. To address the problem, a model-based reinforcement learning algorithm with attention mechanism embedded is proposed as an extension of World Models in this paper. We learn the environment model through Mixture Density Network Recurrent Network(MDN-RNN) for agents to interact, with combinations of variational auto-encoder(VAE) and attention incorporated in state value estimates during the process of learning policy. In this way, agent can learn optimal policies through less interactions with actual environment, and final experiments demonstrate the effectiveness of our model in control problem.
Provable Multi-Objective Reinforcement Learning with Generative Models

Dongruo Zhou,Jiahao Chen,Quanquan Gu

DOI: https://doi.org/10.48550/arXiv.2011.10134

2021-01-11

Abstract:Multi-objective reinforcement learning (MORL) is an extension of ordinary, single-objective reinforcement learning (RL) that is applicable to many real-world tasks where multiple objectives exist without known relative costs. We study the problem of single policy MORL, which learns an optimal policy given the preference of objectives. Existing methods require strong assumptions such as exact knowledge of the multi-objective Markov decision process, and are analyzed in the limit of infinite data and time. We propose a new algorithm called model-based envelop value iteration (EVI), which generalizes the enveloped multi-objective $Q$-learning algorithm in Yang et al., 2019. Our method can learn a near-optimal value function with polynomial sample complexity and linear convergence speed. To the best of our knowledge, this is the first finite-sample analysis of MORL algorithms.

Machine Learning,Artificial Intelligence,Optimization and Control
Reinforcement Learning with Generative Models for Compact Support Sets

Nico Schiavone,Xingyu Li

2024-04-25

Abstract:Foundation models contain a wealth of information from their vast number of training samples. However, most prior arts fail to extract this information in a precise and efficient way for small sample sizes. In this work, we propose a framework utilizing reinforcement learning as a control for foundation models, allowing for the granular generation of small, focused synthetic support sets to augment the performance of neural network models on real data classification tasks. We first allow a reinforcement learning agent access to a novel context based dictionary; the agent then uses this dictionary with a novel prompt structure to form and optimize prompts as inputs to generative models, receiving feedback based on a reward function combining the change in validation accuracy and entropy. A support set is formed this way over several exploration steps. Our framework produced excellent results, increasing classification accuracy by significant margins for no additional labelling or data cost.

Machine Learning,Computer Vision and Pattern Recognition
Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models and Amortized Policy Search

Qi Wang,Herke van Hoof

DOI: https://doi.org/10.48550/arxiv.2102.08291

2022-01-01

Abstract:Reinforcement learning is a promising paradigm for solving sequential decision-making problems, but low data efficiency and weak generalization across tasks are bottlenecks in real-world applications. Model-based meta reinforcement learning addresses these issues by learning dynamics and leveraging knowledge from prior experience. In this paper, we take a closer look at this framework and propose a new posterior sampling based approach that consists of a new model to identify task dynamics together with an amortized policy optimization step. We show that our model, called a graph structured surrogate model (GSSM), achieves competitive dynamics prediction performance with lower model complexity. Moreover, our approach in policy search is able to obtain high returns and allows fast execution by avoiding test-time policy gradient updates.
Structure learning with Temporal Gaussian Mixture for model-based Reinforcement Learning

Théophile Champion,Marek Grześ,Howard Bowman

2024-11-18

Abstract:Model-based reinforcement learning refers to a set of approaches capable of sample-efficient decision making, which create an explicit model of the environment. This model can subsequently be used for learning optimal policies. In this paper, we propose a temporal Gaussian Mixture Model composed of a perception model and a transition model. The perception model extracts discrete (latent) states from continuous observations using a variational Gaussian mixture likelihood. Importantly, our model constantly monitors the collected data searching for new Gaussian components, i.e., the perception model performs a form of structure learning (Smith et al., 2020; Friston et al., 2018; Neacsu et al., 2022) as it learns the number of Gaussian components in the mixture. Additionally, the transition model learns the temporal transition between consecutive time steps by taking advantage of the Dirichlet-categorical conjugacy. Both the perception and transition models are able to forget part of the data points, while integrating the information they provide within the prior, which ensure fast variational inference. Finally, decision making is performed with a variant of Q-learning which is able to learn Q-values from beliefs over states. Empirically, we have demonstrated the model's ability to learn the structure of several mazes: the model discovered the number of states and the transition probabilities between these states. Moreover, using its learned Q-values, the agent was able to successfully navigate from the starting position to the maze's exit.

Machine Learning,Artificial Intelligence
Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

Xueying Bai,Jian Guan,Hongning Wang

DOI: https://doi.org/10.48550/arxiv.1911.03845

2019-01-01

Abstract:Reinforcement learning is well suited for optimizing policies of recommender systems. Current solutions mostly focus on model-free approaches, which require frequent interactions with the real environment, and thus are expensive in model learning. Offline evaluation methods, such as importance sampling, can alleviate such limitations, but usually request a large amount of logged data and do not work well when the action space is large. In this work, we propose a model-based reinforcement learning solution which models user-agent interaction for offline policy learning via a generative adversarial network. To reduce bias in the learned model and policy, we use a discriminator to evaluate the quality of generated data and scale the generated rewards. Our theoretical analysis and empirical evaluations demonstrate the effectiveness of our solution in learning policies from the offline and generated data.
Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model

Gen Li,Yuting Wei,Yuejie Chi,Yuxin Chen

DOI: https://doi.org/10.1287/opre.2023.2451

IF: 2.7

2023-04-23

Operations Research

Abstract:This paper studies a central issue in modern reinforcement learning, the sample efficiency, and makes progress toward solving an idealistic scenario that assumes access to a generative model or a simulator. Despite a large number of prior works tackling this problem, a complete picture of the trade-offs between sample complexity and statistical accuracy has yet to be determined. In particular, all prior results suffer from a severe sample size barrier in the sense that their claimed statistical guarantees hold only when the sample size exceeds some enormous threshold. The current paper overcomes this barrier and fully settles this problem; more specifically, we establish the minimax optimality of the model-based approach for any given target accuracy level. To the best of our knowledge, this work delivers the first minimax-optimal guarantees that accommodate the entire range of sample sizes (beyond which finding a meaningful policy is information theoretically infeasible).

management,operations research & management science
Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning

Junjie Wang,Yao Mu,Dong Li,Qichao Zhang,Dongbin Zhao,Yuzheng Zhuang,Ping Luo,Bin Wang,Jianye Hao

DOI: https://doi.org/10.48550/arXiv.2211.12774

IF: 5.414

2022-11-23

Machine Learning

Abstract:The latent world model provides a promising way to learn policies in a compact latent space for tasks with high-dimensional observations, however, its generalization across diverse environments with unseen dynamics remains challenging. Although the recurrent structure utilized in current advances helps to capture local dynamics, modeling only state transitions without an explicit understanding of environmental context limits the generalization ability of the dynamics model. To address this issue, we propose a Prototypical Context-Aware Dynamics (ProtoCAD) model, which captures the local dynamics by time consistent latent context and enables dynamics generalization in high-dimensional control tasks. ProtoCAD extracts useful contextual information with the help of the prototypes clustered over batch and benefits model-based RL in two folds: 1) It utilizes a temporally consistent prototypical regularizer that encourages the prototype assignments produced for different time parts of the same latent trajectory to be temporally consistent instead of comparing the features; 2) A context representation is designed which combines both the projection embedding of latent states and aggregated prototypes and can significantly improve the dynamics generalization ability. Extensive experiments show that ProtoCAD surpasses existing methods in terms of dynamics generalization. Compared with the recurrent-based model RSSM, ProtoCAD delivers 13.2% and 26.7% better mean and median performance across all dynamics generalization tasks.
Model-based Policy Optimization with Unsupervised Model Adaptation

Jian Shen,Han Zhao,Weinan Zhang,Yong Yu

2020-01-01

Abstract: Model-based reinforcement learning methods learn a dynamics model with real data sampled from the environment and leverage it to generate simulated data to derive an agent. However, due to the potential distribution mismatch between simulated data and real data, this could lead to degraded performance. Despite much effort being devoted to reducing this distribution mismatch, existing methods fail to solve it explicitly. In this paper, we investigate how to bridge the gap between real and simulated data due to inaccurate model estimation for better policy optimization. To begin with, we first derive a lower bound of the expected return, which naturally inspires a bound maximization algorithm by aligning the simulated and real data distributions. To this end, we propose a novel model-based reinforcement learning framework AMPO, which introduces unsupervised model adaptation to minimize the integral probability metric (IPM) between feature distributions from real and simulated data. Instantiating our framework with Wasserstein-1 distance gives a practical model-based approach. Empirically, our approach achieves state-of-the-art performance in terms of sample efficiency on a range of continuous control benchmark tasks.
Towards a Simple Approach to Multi-step Model-based Reinforcement Learning

Kavosh Asadi,Evan Cater,Dipendra Misra,Michael L. Littman

DOI: https://doi.org/10.48550/arXiv.1811.00128

2018-11-01

Abstract:When environmental interaction is expensive, model-based reinforcement learning offers a solution by planning ahead and avoiding costly mistakes. Model-based agents typically learn a single-step transition model. In this paper, we propose a multi-step model that predicts the outcome of an action sequence with variable length. We show that this model is easy to learn, and that the model can make policy-conditional predictions. We report preliminary results that show a clear advantage for the multi-step model compared to its one-step counterpart.

Machine Learning,Artificial Intelligence
Model-Based Off-Policy Deep Reinforcement Learning with Model-Embedding

Xiaoyu Tan,Chao Qu,Junwu Xiong,James Zhang,Xihe Qiu,Yaochu Jin

DOI: https://doi.org/10.1109/tetci.2024.3369636

2024-01-01

IEEE Transactions on Emerging Topics in Computational Intelligence

Abstract:Model-based reinforcement learning (MBRL) has shown its advantages in sample efficiency over model-free reinforcement learning (MFRL) by leveraging control-based domain knowledge. Despite the impressive results it achieves, MBRL is still outperformed by MFRL due to the lack of unlimited interactions with the environment. While imaginary data can be generated by imagining the trajectories of future states, a trade-off between the usage of data generation and the influence of model bias remains to be resolved. In this paper, we propose a simple and elegant off-policy model-based deep reinforcement learning algorithm with a model embedded in the framework of probabilistic reinforcement learning, called MEMB. To balance the sample-efficiency and model bias, we exploit both real and imaginary data in training. In particular, we embed the model in the policy update and learn value functions from the real data set. We also provide a theoretical analysis of MEMB with the Lipschitz continuity assumption on the model and policy, proving the reliability of the short-term imaginary rollout. Finally, we evaluate MEMB on several benchmarks and demonstrate that our algorithm can achieve state-of-the-art performance.
Model-Based Reinforcement Learning Via Imagination with Derived Memory.

Yao Mu,Yuzheng Zhuang,Bin Wang,Guangxiang Zhu,Wulong Liu,Jianyu Chen,Ping Luo,Shengbo Eben Li,Chongjie Zhang,Jianye Hao

2021-01-01

Abstract:Model-based reinforcement learning aims to improve the sample efficiency of policy learning by modelling the dynamics of the environment. Recently, the latent dynamics model has been further developed to enable fast planning in a compact space. It summarizes the high-dimensional experiences of an agent, which mimics the memory function of humans. Learning policies via imagination with the latent model shows great potential for solving complex tasks. However, only considering memories from the true experiences in the process of imagination could limit its advantages. Inspired by the memory prosthesis proposed by neuroscientists, we present a novel model-based reinforcement learning framework called Imagining with Derived Memory (IDM). It enables the agent to learn policy from enriched diverse imagination with prediction-reliability weight, thus improving sample efficiency and policy robustness. Experiments on various high-dimensional visual control tasks in the DMControl benchmark demonstrate that IDM outperforms previous state-of-the-art methods in terms of policy robustness and further improves the sample efficiency of the model-based method.
Generative Adversarial User Model for Reinforcement Learning Based Recommendation System

Xinshi Chen,Shuang Li,Hui Li,Shaohua Jiang,Yuan Qi,Le Song

DOI: https://doi.org/10.48550/arxiv.1812.10613

2019-01-01

Abstract:There are great interests as well as many challenges in applying reinforcement learning (RL) to recommendation systems. In this setting, an online user is the environment; neither the reward function nor the environment dynamics are clearly defined, making the application of RL challenging. In this paper, we propose a novel model-based reinforcement learning framework for recommendation systems, where we develop a generative adversarial network to imitate user behavior dynamics and learn her reward function. Using this user model as the simulation environment, we develop a novel Cascading DQN algorithm to obtain a combinatorial recommendation policy which can handle a large number of candidate items efficiently. In our experiments with real data, we show this generative adversarial user model can better explain user behavior than alternatives, and the RL policy based on this model can lead to a better long-term reward for the user and higher click rate for the system.
Value Gradient weighted Model-Based Reinforcement Learning

Claas Voelcker,Victor Liao,Animesh Garg,Amir-massoud Farahmand

2023-06-21

Abstract:Model-based reinforcement learning (MBRL) is a sample efficient technique to obtain control policies, yet unavoidable modeling errors often lead performance deterioration. The model in MBRL is often solely fitted to reconstruct dynamics, state observations in particular, while the impact of model error on the policy is not captured by the training objective. This leads to a mismatch between the intended goal of MBRL, enabling good policy and value learning, and the target of the loss function employed in practice, future state prediction. Naive intuition would suggest that value-aware model learning would fix this problem and, indeed, several solutions to this objective mismatch problem have been proposed based on theoretical analysis. However, they tend to be inferior in practice to commonly used maximum likelihood (MLE) based approaches. In this paper we propose the Value-gradient weighted Model Learning (VaGraM), a novel method for value-aware model learning which improves the performance of MBRL in challenging settings, such as small model capacity and the presence of distracting state dimensions. We analyze both MLE and value-aware approaches and demonstrate how they fail to account for exploration and the behavior of function approximation when learning value-aware models and highlight the additional goals that must be met to stabilize optimization in the deep learning setting. We verify our analysis by showing that our loss function is able to achieve high returns on the Mujoco benchmark suite while being more robust than maximum likelihood based approaches.

Machine Learning,Artificial Intelligence

A model-based reinforcement learning method based on conditional generative adversarial networks

Model Gradient: Unified Model and Policy Learning in Model-Based Reinforcement Learning

Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective

Model-Based Reinforcement Learning via Meta-Policy Optimization

Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference

Intelligent Model Learning Based on Variance for Bayesian Reinforcement Learning

VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control

Provable Multi-Objective Reinforcement Learning with Generative Models

Reinforcement Learning with Generative Models for Compact Support Sets

Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models and Amortized Policy Search

Structure learning with Temporal Gaussian Mixture for model-based Reinforcement Learning

Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model

Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning

Model-based Policy Optimization with Unsupervised Model Adaptation

Towards a Simple Approach to Multi-step Model-based Reinforcement Learning

Model-Based Off-Policy Deep Reinforcement Learning with Model-Embedding

Model-Based Reinforcement Learning Via Imagination with Derived Memory.

Generative Adversarial User Model for Reinforcement Learning Based Recommendation System

Value Gradient weighted Model-Based Reinforcement Learning