Abstract:Machines that can replicate human intelligence with type 2 reasoning capabilities should be able to reason at multiple levels of spatio-temporal abstractions and scales using internal world models. Devising formalisms to develop such internal world models, which accurately reflect the causal hierarchies inherent in the dynamics of the real world, is a critical research challenge in the domains of artificial intelligence and machine learning. This thesis identifies several limitations with the prevalent use of state space models (SSMs) as internal world models and propose two new probabilistic formalisms namely Hidden-Parameter SSMs and Multi-Time Scale SSMs to address these drawbacks. The structure of graphical models in both formalisms facilitates scalable exact probabilistic inference using belief propagation, as well as end-to-end learning via backpropagation through time. This approach permits the development of scalable, adaptive hierarchical world models capable of representing nonstationary dynamics across multiple temporal abstractions and scales. Moreover, these probabilistic formalisms integrate the concept of uncertainty in world states, thus improving the system's capacity to emulate the stochastic nature of the real world and quantify the confidence in its predictions. The thesis also discuss how these formalisms are in line with related neuroscience literature on Bayesian brain hypothesis and predicitive processing. Our experiments on various real and simulated robots demonstrate that our formalisms can match and in many cases exceed the performance of contemporary transformer variants in making long-range future predictions. We conclude the thesis by reflecting on the limitations of our current models and suggesting directions for future research.

Learning Discrete State Abstractions With Deep Variational Inference

Learning Markov State Abstractions for Deep Reinforcement Learning

Learning in Hybrid Active Inference Models

Learning Invariant Representations for Reinforcement Learning without Reconstruction

Deep Active Inference as Variational Policy Gradients

Variational Temporal Abstraction

Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

State Abstraction via Deep Supervised Hash Learning

Learning with Language-Guided State Abstractions

Learning Dynamic Abstract Representations for Sample-Efficient Reinforcement Learning

Deep Interactive Bayesian Reinforcement Learning via Meta-Learning

PcLast: Discovering Plannable Continuous Latent States

Contrastive Abstraction for Reinforcement Learning

Learning World Models With Hierarchical Temporal Abstractions: A Probabilistic Perspective

Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction

Learning Action-based Representations Using Invariance

Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning

Variational learning for switching state-space models

Learning Non-Markovian Decision-Making from State-only Sequences

Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments

Deconfounding Imitation Learning with Variational Inference