Abstract:Model-based reinforcement learning (MBRL) holds the promise of sample-efficient learning by utilizing a world model, which models how the environment works and typically encompasses components for two tasks: observation modeling and reward modeling. In this paper, through a dedicated empirical investigation, we gain a deeper understanding of the role each task plays in world models and uncover the overlooked potential of sample-efficient MBRL by mitigating the domination of either observation or reward modeling. Our key insight is that while prevalent approaches of explicit MBRL attempt to restore abundant details of the environment via observation models, it is difficult due to the environment's complexity and limited model capacity. On the other hand, reward models, while dominating implicit MBRL and adept at learning compact task-centric dynamics, are inadequate for sample-efficient learning without richer learning signals. Motivated by these insights and discoveries, we propose a simple yet effective approach, HarmonyDream, which automatically adjusts loss coefficients to maintain task harmonization, i.e. a dynamic equilibrium between the two tasks in world model learning. Our experiments show that the base MBRL method equipped with HarmonyDream gains 10%-69% absolute performance boosts on visual robotic tasks and sets a new state-of-the-art result on the Atari 100K benchmark. Code is available at <a class="link-external link-https" href="https://github.com/thuml/HarmonyDream" rel="external noopener nofollow">this https URL</a>.

Dr. Strategy: Model-Based Generalist Agents with Strategic Dreaming

MA-Dreamer: Coordination and communication through shared imagination

DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations

TransDreamer: Reinforcement Learning with Transformer World Models

DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation

Towards biologically plausible Dreaming and Planning in recurrent spiking networks

Sample-efficient multi-agent reinforcement learning with masked reconstruction

Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

Dream to Control: Learning Behaviors by Latent Imagination

Policy-shaped prediction: avoiding distractions in model-based reinforcement learning

SafeDreamer: Safe Reinforcement Learning with World Models

Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalization

HarmonyDream: Task Harmonization Inside World Models

Model-based Exploration Strategy to Accelerate Deterministic Strategy Algorithm Training

A Human Mixed Strategy Approach to Deep Reinforcement Learning

Dream to Adapt: Meta Reinforcement Learning by Latent Context Imagination and MDP Imagination

Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning

MuDreamer: Learning Predictive World Models without Reconstruction

Maximum Entropy Model-based Reinforcement Learning

Learning Latent Dynamic Robust Representations for World Models