Abstract:Learning to reach goal states and learning diverse skills through mutual information (MI) maximization have been proposed as principled frameworks for self-supervised reinforcement learning, allowing agents to acquire broadly applicable multitask policies with minimal reward engineering. Starting from a simple observation that the standard goal-conditioned RL (GCRL) is encapsulated by the optimization objective of variational empowerment, we discuss how GCRL and MI-based RL can be generalized into a single family of methods, which we name variational GCRL (VGCRL), interpreting variational MI maximization, or variational empowerment, as representation learning methods that acquire functionally-aware state representations for goal reaching. This novel perspective allows us to: (1) derive simple but unexplored variants of GCRL to study how adding small representation capacity can already expand its capabilities; (2) investigate how discriminator function capacity and smoothness determine the quality of discovered skills, or latent goals, through modifying latent dimensionality and applying spectral normalization; (3) adapt techniques such as hindsight experience replay (HER) from GCRL to MI-based RL; and lastly, (4) propose a novel evaluation metric, named latent goal reaching (LGR), for comparing empowerment algorithms with different choices of latent dimensionality and discriminator parameterization. Through principled mathematical derivations and careful experimental studies, our work lays a novel foundation from which to evaluate, analyze, and develop representation learning techniques in goal-based RL.

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

Learning Efficient Representation for Intrinsic Motivation

On Variational Bounds of Mutual Information

An Information Theoretic Approach to Interaction-Grounded Learning

A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning

Discovering Influential Factors in Variational Autoencoder

A robust estimator of mutual information for deep learning interpretability

INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL

Variational Intrinsic Control

Infrared and Visible Image Fusion Via Mutual Information Maximization

Learning Representations in Reinforcement Learning:An Information Bottleneck Approach

Mutual information maximization for amortized likelihood inference from sampled trajectories: MINIMALIST

Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses

Visualizing Information Bottleneck through Variational Inference

Supposed Maximum Mutual Information for Improving Generalization and Interpretation of Multi-Layered Neural Networks

Mutual Information Gradient Estimation for Representation Learning

Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning

Information is Power: Intrinsic Control via Information Capture

Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS