Curriculum offline reinforcement learning

Yuanying Cai, Chuheng Zhang, Hanye Zhao, Li Zhao, Jiang Bian

2023-05-30

Abstract:Despite the great success and popularity gained by reinforcement learning (RL)[49], there exists a wide range of real-world applications that can hardly be solved using RL due to the high cost or the risk of online interactions, such as robotics [20], autonomous

What problem does this paper attempt to address?

Curriculum Offline Reinforcement Learning

Yuanying Cai,Chuheng Zhang,Hanye Zhao,Li Zhao,Jiang Bian

2023-01-01

Abstract:Offline reinforcement learning holds the promise of obtaining powerful agents from large datasets. To achieve this, a good algorithm should always benefit from (or at least does not degenerate by) adding more samples, even if the samples are not collected by expert policies. However, we observe that many popular offline RL algorithms do not possess such a property and sometimes suffers from adding heterogeneous or poor samples to the dataset. Empirically we show that, given a stage in the learning process, not all samples are useful for these algorithms. Specifically, the agent can learn more efficiently with only the samples collected by a policy similar to the current policy. This indicates that different samples may contribute to different stages of the training process, and therefore we propose Curriculum Offline Reinforcement Learning (CUORL) to equip the previous methods with the such a favorable property. In CUORL, we select the samples that are likely to be generated by the current policy to train the agent. Empirically, we show that CUORL can prevent the negative impact of adding the samples from poor policies and always improves the performance with more samples (even from random policies). Moreover, CUORL also achieves state-of-the-art performance on standard D4RL datasets, which indicates the potential of curriculum learning for offline RL.
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

Rafael Figueiredo Prudencio,Marcos R. O. A. Maximo,Esther Luna Colombini

DOI: https://doi.org/10.1109/TNNLS.2023.3250269

2023-04-19

Abstract:With the widespread adoption of deep learning, reinforcement learning (RL) has experienced a dramatic increase in popularity, scaling to previously intractable problems, such as playing complex games from pixel observations, sustaining conversations with humans, and controlling robotic agents. However, there is still a wide range of domains inaccessible to RL due to the high cost and danger of interacting with the environment. Offline RL is a paradigm that learns exclusively from static datasets of previously collected interactions, making it feasible to extract policies from large and diverse training datasets. Effective offline RL algorithms have a much wider range of applications than online RL, being particularly appealing for real-world applications, such as education, healthcare, and robotics. In this work, we contribute with a unifying taxonomy to classify offline RL methods. Furthermore, we provide a comprehensive review of the latest algorithmic breakthroughs in the field using a unified notation as well as a review of existing benchmarks' properties and shortcomings. Additionally, we provide a figure that summarizes the performance of each method and class of methods on different dataset properties, equipping researchers with the tools to decide which type of algorithm is best suited for the problem at hand and identify which classes of algorithms look the most promising. Finally, we provide our perspective on open problems and propose future research directions for this rapidly growing field.

Machine Learning,Artificial Intelligence
Curriculum Goal-Conditioned Imitation for Offline Reinforcement Learning

Xiaoyun Feng,Li Jiang,Xudong Yu,Haoran Xu,Xiaoyan Sun,Jie Wang,Xianyuan Zhan,Wai Kin Chan

DOI: https://doi.org/10.1109/tg.2022.3224088

IF: 1.237

2022-01-01

IEEE Transactions on Games

Abstract:Offline reinforcement learning (RL) enables learning policies from precollected datasets without online data collection. Although it offers the possibility to surpass the performance of the datasets, most existing offline RL algorithms struggle to compete with behavior cloning policies in many dataset settings due to trading off policy improvement and additional regularization to address the distributional shift issue. In many cases, if one can imitate a sequence of suboptimal subtrajectories in data and properly "stitch" them toward reaching an ideal future state, it may potentially result in a more reliable policy while avoiding difficulties that present in typical value-based offline RL algorithms. We borrow the idea of curriculum learning to embody the above intuition. We construct a curriculum that progressively imitates a sequence of suboptimal trajectories conditioned on a series of carefully constructed future states and cumulative rewards as goals. The suboptimal trajectories gradually guide policy learning toward reaching the ideal goal states. We name our algorithm curriculum goal-conditioned imitation (CGI). Experimental results show that CGI achieves competitive performance against state-of-the-art offline RL algorithms, especially for challenging tasks with long horizons and sparse rewards.
Doubly Constrained Offline Reinforcement Learning for Learning Path Recommendation

Yue Yun,Huan Dai,Rui An,Yupei Zhang,Xuequn Shang

DOI: https://doi.org/10.1016/j.knosys.2023.111242

IF: 8.139

2024-01-01

Knowledge-Based Systems

Abstract:Learning path recommendation refers to the application of interactive recommendation systems in the field of education, aimed at optimizing learning outcomes while minimizing the workload of learners, teachers, and curriculum designers. Reinforcement Learning (RL) has proven effective in capturing and modeling the complex interactions among course activities, learner behaviors, and educational outcomes. Therefore, combining the two approaches presents endless possibilities for personalized education through the use of interactive recommendation systems in the education domain. However, traditional RL algorithms require extensive interaction with the environment during the training phase. Using unverified recommendation logic in interactions with actual students can give rise to unmanageable problems and hinder effective performance in an educational setting. This is because extrapolation introduces substantial evaluation errors that result in recommendations deviating significantly from the actual educational requirements. To address this limitation, we propose a novel method of offline reinforcement learning called Doubly Constrained deep Q-learning Network (DCQN). This method utilizes two generative models to fit existing student historical interaction data, which in turn, constrains the original policy network to generate new actions based on past interactions, avoiding the occurrence of overestimated actions and reducing extrapolation errors. Empirical results on demonstrate that this approach performs better than existing techniques across D4RL, i.e., datasets for deep data-driven reinforcement learning and real educational datasets.
Curriculum-RL Based Air Combat Decision-Making

Yuhang He,Dapeng Yang,Man Zhang,Yan Li

DOI: https://doi.org/10.1007/978-981-19-6613-2_447

2023-01-01

Abstract:Deep reinforcement learning has been able to solve a wide range of complex decision-making tasks that were previously out of reach for a machine. But there are many challenges in using reinforcement learning to solve air combat decision-making problems. One of them is that rewards are sparse in the air combat and most of today’s DRL algorithms struggle with such sparsity. Therefore, we propose a new method called Curriculum-RL which guide agent to learn a task from the simple to the complex. While solving the sparse problem, Curriculum-RL helps the agent to learn the air combat strategy better than the general weighted reward structure. In the end, we test our approach in the two-plane air combat and analyze the result.
Curriculum Offline Imitation Learning

Minghuan Liu,Hanye Zhao,Zhengyu Yang,Jian Shen,Weinan Zhang,Li Zhao,Tie-Yan Liu

DOI: https://doi.org/10.48550/arxiv.2111.02056

2021-01-01

Abstract:Offline reinforcement learning (RL) tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. Despite the potential to surpass the behavioral policies, RL-based methods are generally impractical due to the training instability and bootstrapping the extrapolation errors, which always require careful hyperparameter tuning via online evaluation. In contrast, offline imitation learning (IL) has no such issues since it learns the policy directly without estimating the value function by bootstrapping. However, IL is usually limited in the capability of the behavioral policy and tends to learn a mediocre behavior from the dataset collected by the mixture of policies. In this paper, we aim to take advantage of IL but mitigate such a drawback. Observing that behavior cloning is able to imitate neighboring policies with less data, we propose Curriculum Offline Imitation Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return, and improves the current policy along curriculum stages. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
Reward-free Offline Reinforcement Learning

Zhenbo Huang,Shiliang Sun,Jing Zhao

DOI: https://doi.org/10.1016/j.knosys.2024.112018

IF: 8.139

2024-01-01

Knowledge-Based Systems

Abstract:Offline reinforcement learning (RL) aims to learn a policy from pre-collected data, avoiding costly or risky interactions with the environment. In the offline setting, the inherent problem of distribution shift leads to extrapolation error, resulting in policy learning failures. Conventional offline RL methods tackle this by reducing the value estimates of unseen actions or incorporating policy constraints. However, these methods confine the agent’s actions within the data manifold, hampering the agent’s capacity to acquire fresh insights from actions beyond the dataset’s scope. To address this, we propose a novel offline RL method incorporating action exploration, called EoRL. We partition policy learning into behavior and exploration learning, where exploration learning empowers the agent to discover novel actions, while behavior learning approximates the behavior policy. Specifically, in exploratory learning, we define the deviation between decision actions and dataset actions as the action novelty, replacing the traditional reward with an assessment of the cumulative novelty of the policy. Additionally, behavior policy restricts actions to the vicinity of the dataset-supported actions, and the two parts of the policy learning share parameters. We demonstrate EoRL’s ability to explore a larger action space while controlling the policy shift. And its reward-free learning model is more compatible with realistic task scenarios. Experimental results demonstrate the outstanding performance of our method on Mujoco locomotion and 2D maze tasks.
Deploying Offline Reinforcement Learning with Human Feedback

Ziniu Li,Ke Xu,Liu,Lanqing Li,Deheng Ye,Peilin Zhao

DOI: https://doi.org/10.48550/arxiv.2303.07046

2023-01-01

Abstract:Reinforcement learning (RL) has shown promise for decision-making tasks in real-world applications. One practical framework involves training parameterized policy models from an offline dataset and subsequently deploying them in an online environment. However, this approach can be risky since the offline training may not be perfect, leading to poor performance of the RL models that may take dangerous actions. To address this issue, we propose an alternative framework that involves a human supervising the RL models and providing additional feedback in the online deployment phase. We formalize this online deployment problem and develop two approaches. The first approach uses model selection and the upper confidence bound algorithm to adaptively select a model to deploy from a candidate set of trained offline RL models. The second approach involves fine-tuning the model in the online deployment phase when a supervision signal arrives. We demonstrate the effectiveness of these approaches for robot locomotion control and traffic light control tasks through empirical validation.
Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

Anikait Singh,Aviral Kumar,Quan Vuong,Yevgen Chebotar,Sergey Levine

DOI: https://doi.org/10.48550/arXiv.2211.01052

2022-11-21

Abstract:Offline reinforcement learning (RL) learns policies entirely from static datasets, thereby avoiding the challenges associated with online data collection. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto a highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, we show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy to the same extent across the state space. Ideally, the learned policy should be free to choose per state how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution in conservative Q-learning (CQL) to obtain an approximate support constraint formulation. The reweighted distribution is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.

Machine Learning
Semi-supervised reward learning for offline reinforcement learning

Ksenia Konyushkova,Konrad Zolna,Yusuf Aytar,Alexander Novikov,Scott Reed,Serkan Cabi,Nando de Freitas

DOI: https://doi.org/10.48550/arXiv.2012.06899

2020-12-13

Abstract:In offline reinforcement learning (RL) agents are trained using a logged dataset. It appears to be the most natural route to attack real-life applications because in domains such as healthcare and robotics interactions with the environment are either expensive or unethical. Training agents usually requires reward functions, but unfortunately, rewards are seldom available in practice and their engineering is challenging and laborious. To overcome this, we investigate reward learning under the constraint of minimizing human reward annotations. We consider two types of supervision: timestep annotations and demonstrations. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards. We further investigate the relationship between the quality of the reward model and the final policies. We notice, for example, that the reward models do not need to be perfect to result in useful policies.

Machine Learning,Artificial Intelligence,Robotics
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

Yiqin Yang,Xiaoteng Ma,Chenghao Li,Zewu Zheng,Qiyuan Zhang,Gao Huang,Jun Yang,Qianchuan Zhao

DOI: https://doi.org/10.48550/arxiv.2106.03400

2021-01-01

Abstract:Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios. However, compared with the single-agent counterpart, offline multi-agent RL introduces more agents with the larger state and action space, which is more challenging but attracts little attention. We demonstrate current offline RL algorithms are ineffective in multi-agent systems due to the accumulated extrapolation error. In this paper, we propose a novel offline RL algorithm, named Implicit Constraint Q-learning (ICQ), which effectively alleviates the extrapolation error by only trusting the state-action pairs given in the dataset for value estimation. Moreover, we extend ICQ to multi-agent tasks by decomposing the joint-policy under the implicit constraint. Experimental results demonstrate that the extrapolation error is successfully controlled within a reasonable range and insensitive to the number of agents. We further show that ICQ achieves the state-of-the-art performance in the challenging multi-agent offline tasks (StarCraft II). Our code is public online at https://github.com/YiqinYang/ICQ.
Curriculum offline imitating learning

Minghuan Liu, Hanye Zhao, Zhengyu Yang, Jian Shen, Weinan Zhang, Li Zhao, Tie-Yan Liu

2021-12-06

Abstract:Offline reinforcement learning (RL) tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. Despite the potential to surpass the behavioral policies, RL-based methods are generally impractical due to the training instability and bootstrapping the extrapolation errors, which always require careful hyperparameter tuning via online evaluation. In contrast, offline imitation learning (IL) has no such issues since it learns the policy directly without estimating the value function by bootstrapping. However, IL is usually limited in the capability of the behavioral policy and tends to learn a mediocre behavior from the dataset collected by the mixture of policies. In this paper, we aim to take advantage of IL but mitigate such a drawback. Observing that behavior cloning is able to imitate neighboring policies with less data, we propose\textit {Curriculum Offline Imitation Learning (COIL)}, which utilizes an experience picking strategy to make the agent imitate from adaptive neighboring policies with a higher return, and improves the current policy along curriculum stages. On continuous control benchmarks, we compare COIL against both imitation-based methods and RL-based methods, showing that COIL not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
Offline Reinforcement Learning with Imbalanced Datasets

Li Jiang,Sijie Cheng,Jielin Qiu,Haoran Xu,Wai Kin Chan,Zhao Ding

2024-05-21

Abstract:The prevalent use of benchmarks in current offline reinforcement learning (RL) research has led to a neglect of the imbalance of real-world dataset distributions in the development of models. The real-world offline RL dataset is often imbalanced over the state space due to the challenge of exploration or safety considerations. In this paper, we specify properties of imbalanced datasets in offline RL, where the state coverage follows a power law distribution characterized by skewed policies. Theoretically and empirically, we show that typically offline RL methods based on distributional constraints, such as conservative Q-learning (CQL), are ineffective in extracting policies under the imbalanced dataset. Inspired by natural intelligence, we propose a novel offline RL method that utilizes the augmentation of CQL with a retrieval process to recall past related experiences, effectively alleviating the challenges posed by imbalanced datasets. We evaluate our method on several tasks in the context of imbalanced datasets with varying levels of imbalance, utilizing the variant of D4RL. Empirical results demonstrate the superiority of our method over other baselines.

Machine Learning,Artificial Intelligence
Towards Data-Driven Offline Simulations for Online Reinforcement Learning

Shengpu Tang,Felipe Vieira Frujeri,Dipendra Misra,Alex Lamb,John Langford,Paul Mineiro,Sebastian Kochman

DOI: https://doi.org/10.48550/arXiv.2211.07614

2022-11-15

Abstract:Modern decision-making systems, from robots to web recommendation engines, are expected to adapt: to user preferences, changing circumstances or even new tasks. Yet, it is still uncommon to deploy a dynamically learning agent (rather than a fixed policy) to a production system, as it's perceived as unsafe. Using historical data to reason about learning algorithms, similar to offline policy evaluation (OPE) applied to fixed policies, could help practitioners evaluate and ultimately deploy such adaptive agents to production. In this work, we formalize offline learner simulation (OLS) for reinforcement learning (RL) and propose a novel evaluation protocol that measures both fidelity and efficiency of the simulation. For environments with complex high-dimensional observations, we propose a semi-parametric approach that leverages recent advances in latent state discovery in order to achieve accurate and efficient offline simulations. In preliminary experiments, we show the advantage of our approach compared to fully non-parametric baselines. The code to reproduce these experiments will be made available at <a class="link-external link-https" href="https://github.com/microsoft/rl-offline-simulation" rel="external noopener nofollow">this https URL</a>.

Machine Learning
When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?

Aviral Kumar,Joey Hong,Anikait Singh,Sergey Levine

DOI: https://doi.org/10.48550/arXiv.2204.05618

2022-04-12

Abstract:Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even from highly suboptimal data, a scenario where imitation learning finds suboptimal solutions that do not improve over the demonstrator that generated the dataset. However, another common use case for practitioners is to learn from data that resembles demonstrations. In this case, one can choose to apply offline RL, but can also use behavioral cloning (BC) algorithms, which mimic a subset of the dataset via supervised learning. Therefore, it seems natural to ask: when can an offline RL method outperform BC with an equal amount of expert data, even when BC is a natural choice? To answer this question, we characterize the properties of environments that allow offline RL methods to perform better than BC methods, even when only provided with expert data. Additionally, we show that policies trained on sufficiently noisy suboptimal data can attain better performance than even BC algorithms with expert data, especially on long-horizon problems. We validate our theoretical results via extensive experiments on both diagnostic and high-dimensional domains including robotic manipulation, maze navigation, and Atari games, with a variety of data distributions. We observe that, under specific but common conditions such as sparse rewards or noisy data sources, modern offline RL methods can significantly outperform BC.

Machine Learning
Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement

Tianyu Shi,Dong Chen,Kaian Chen,Zhaojian Li

DOI: https://doi.org/10.48550/arXiv.2110.07067

2021-11-02

Abstract:Reinforcement learning (RL) is a powerful data-driven control method that has been largely explored in autonomous driving tasks. However, conventional RL approaches learn control policies through trial-and-error interactions with the environment and therefore may cause disastrous consequences such as collisions when testing in real-world traffic. Offline RL has recently emerged as a promising framework to learn effective policies from previously-collected, static datasets without the requirement of active interactions, making it especially appealing for autonomous driving applications. Despite promising, existing offline RL algorithms such as Batch-Constrained deep Q-learning (BCQ) generally lead to rather conservative policies with limited exploration efficiency. To address such issues, this paper presents an enhanced BCQ algorithm by employing a learnable parameter noise scheme in the perturbation model to increase the diversity of observed actions. In addition, a Lyapunov-based safety enhancement strategy is incorporated to constrain the explorable state space within a safe region. Experimental results in highway and parking traffic scenarios show that our approach outperforms the conventional RL method, as well as state-of-the-art offline RL algorithms.

Robotics,Systems and Control
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Sergey Levine,Aviral Kumar,George Tucker,Justin Fu

DOI: https://doi.org/10.48550/arXiv.2005.01643

IF: 5.414

2020-05-04

Machine Learning

Abstract:In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.
Interpretable performance analysis towards offline reinforcement learning: A dataset perspective

Chenyang Xi,Bo Tang,Jiajun Shen,Xinfu Liu,Feiyu Xiong,Xueying Li

DOI: https://doi.org/10.48550/arXiv.2105.05473

IF: 5.414

2021-05-12

Machine Learning

Abstract:Offline reinforcement learning (RL) has increasingly become the focus of the artificial intelligent research due to its wide real-world applications where the collection of data may be difficult, time-consuming, or costly. In this paper, we first propose a two-fold taxonomy for existing offline RL algorithms from the perspective of exploration and exploitation tendency. Secondly, we derive the explicit expression of the upper bound of extrapolation error and explore the correlation between the performance of different types of algorithms and the distribution of actions under states. Specifically, we relax the strict assumption on the sufficiently large amount of state-action tuples. Accordingly, we provably explain why batch constrained Q-learning (BCQ) performs better than other existing techniques. Thirdly, after identifying the weakness of BCQ on dataset of low mean episode returns, we propose a modified variant based on top return selection mechanism, which is proved to be able to gain state-of-the-art performance on various datasets. Lastly, we create a benchmark platform on the Atari domain, entitled RL easy go (RLEG), at an estimated cost of more than 0.3 million dollars. We make it open-source for fair and comprehensive competitions between offline RL algorithms with complete datasets and checkpoints being provided.
Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning

Qi Wang,Junming Yang,Yunbo Wang,Xin Jin,Wenjun Zeng,Xiaokang Yang

2024-10-29

Abstract:Training offline RL models using visual inputs poses two significant challenges, i.e., the overfitting problem in representation learning and the overestimation bias for expected future rewards. Recent work has attempted to alleviate the overestimation bias by encouraging conservative behaviors. This paper, in contrast, tries to build more flexible constraints for value estimation without impeding the exploration of potential advantages. The key idea is to leverage off-the-shelf RL simulators, which can be easily interacted with in an online manner, as the "test bed" for offline policies. To enable effective online-to-offline knowledge transfer, we introduce CoWorld, a model-based RL approach that mitigates cross-domain discrepancies in state and reward spaces. Experimental results demonstrate the effectiveness of CoWorld, outperforming existing RL approaches by large margins.

Machine Learning
Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Han Zheng,Xufang Luo,Pengfei Wei,Xuan Song,Dongsheng Li,Jing Jiang

DOI: https://doi.org/10.48550/arXiv.2303.07693

2023-03-14

Abstract:Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when online interactions are costly. Offline RL provides an alternative solution by directly learning from the previously collected dataset. However, it will yield unsatisfactory performance if the quality of the offline datasets is poor. In this paper, we consider an offline-to-online setting where the agent is first learned from the offline dataset and then trained online, and propose a framework called Adaptive Policy Learning for effectively taking advantage of offline and online data. Specifically, we explicitly consider the difference between the online and offline data and apply an adaptive update scheme accordingly, that is, a pessimistic update strategy for the offline dataset and an optimistic/greedy update scheme for the online dataset. Such a simple and effective method provides a way to mix the offline and online RL and achieve the best of both worlds. We further provide two detailed algorithms for implementing the framework through embedding value or policy-based RL algorithms into it. Finally, we conduct extensive experiments on popular continuous control tasks, and results show that our algorithm can learn the expert policy with high sample efficiency even when the quality of offline dataset is poor, e.g., random dataset.

Machine Learning,Artificial Intelligence

Curriculum offline reinforcement learning

Curriculum Offline Reinforcement Learning

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

Curriculum Goal-Conditioned Imitation for Offline Reinforcement Learning

Doubly Constrained Offline Reinforcement Learning for Learning Path Recommendation

Curriculum-RL Based Air Combat Decision-Making

Curriculum Offline Imitation Learning

Reward-free Offline Reinforcement Learning

Deploying Offline Reinforcement Learning with Human Feedback

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

Semi-supervised reward learning for offline reinforcement learning

Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

Curriculum offline imitating learning

Offline Reinforcement Learning with Imbalanced Datasets

Towards Data-Driven Offline Simulations for Online Reinforcement Learning

When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?

Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancement

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Interpretable performance analysis towards offline reinforcement learning: A dataset perspective

Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning