Abstract:We propose denoising diffusion models for data-driven representation learning of dynamical systems. In this type of generative deep learning, a neural network is trained to denoise and reverse a diffusion process, where Gaussian noise is added to states from the attractor of a dynamical system. Iteratively applied, the neural network can then map samples from isotropic Gaussian noise to the state distribution. We showcase the potential of such neural networks in proof-of-concept experiments with the Lorenz 1963 system. Trained for state generation, the neural network can produce samples that are almost indistinguishable from those on the attractor. The model has thereby learned an internal representation of the system, applicable for different tasks other than state generation. As a first task, we fine-tune the pre-trained neural network for surrogate modelling by retraining its last layer and keeping the remaining network as a fixed feature extractor. In these low-dimensional settings, such fine-tuned models perform similarly to deep neural networks trained from scratch. As a second task, we apply the pre-trained model to generate an ensemble out of a deterministic run. Diffusing the run, and then iteratively applying the neural network, conditions the state generation, which allows us to sample from the attractor in the run's neighbouring region. To control the resulting ensemble spread and Gaussianity, we tune the diffusion time and, thus, the sampled portion of the attractor. While easier to tune, this proposed ensemble sampler can outperform tuned static covariances in ensemble optimal interpolation. Therefore, these two applications show that denoising diffusion models are a promising way towards representation learning for dynamical systems.

SODA: Bottleneck Diffusion Models for Representation Learning

Diffusion Models and Representation Learning: A Survey

Do text-free diffusion models learn discriminative visual representations?

Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

Representation Learning with Diffusion Models

Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts

Bridging Generative and Discriminative Models for Unified Visual Perception with Diffusion Priors

Diffusion-Based Representation Learning

Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

Label-Efficient Semantic Segmentation with Diffusion Models

DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery.

Object-Centric Slot Diffusion

SOD‐diffusion: Salient Object Detection via Diffusion‐Based Image Generators

Representation learning with unconditional denoising diffusion models for dynamical systems

RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation

Diffusion Model as Representation Learner

Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think

Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions

$\textit{Revelio}$: Interpreting and leveraging semantic information in diffusion models

Diffusion Models in Vision: A Survey