Abstract:Multimodal variational autoencoders (VAEs) aim to capture shared latent representations by integrating information from different data modalities. A significant challenge is accurately inferring representations from any subset of modalities without training an impractical number (2^M) of inference networks for all possible modality combinations. Mixture-based models simplify this by requiring only as many inference models as there are modalities, aggregating unimodal inferences. However, they suffer from information loss when modalities are missing. Alignment-based VAEs address this by aligning unimodal inference models with a multimodal model through minimizing the Kullback-Leibler (KL) divergence but face issues due to amortization gaps, which compromise inference accuracy. To tackle these problems, we introduce multimodal iterative amortized inference, an iterative refinement mechanism within the multimodal VAE framework. This method overcomes information loss from missing modalities and minimizes the amortization gap by iteratively refining the multimodal inference using all available modalities. By aligning unimodal inference to this refined multimodal posterior, we achieve unimodal inferences that effectively incorporate multimodal information while requiring only unimodal inputs during inference. Experiments on benchmark datasets show that our approach improves inference performance, evidenced by higher linear classification accuracy and competitive cosine similarity, and enhances cross-modal generation, indicated by lower FID scores. This demonstrates that our method enhances inferred representations from unimodal inputs.

MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning

Audio-Visual Cross-Modal Generation with Multimodal Variational Generative Model

Leveraging hierarchy in multimodal generative models for effective cross-modality inference

Multimodal Generative Models for Scalable Weakly-Supervised Learning

Joint Multimodal Learning with Deep Generative Models

Multimodal Variational Autoencoder: a Barycentric View

Generalizing Multimodal Variational Methods to Sets

Multimodal Generative Models for Compositional Representation Learning

Improving Bi-directional Generation between Different Modalities with Variational Autoencoders

Unified Cross-Modal Image Synthesis with Hierarchical Mixture of Product-of-Experts

Multimodal hierarchical Variational AutoEncoders with Factor Analysis latent space

A Markov Random Field Multi-Modal Variational AutoEncoder

HVAE: A Deep Generative Model Via Hierarchical Variational Auto-Encoder for Multi-View Document Modeling

Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations

Enhancing Unimodal Latent Representations in Multimodal VAEs through Iterative Amortized Inference

Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives

Learning Multimodal VAEs through Mutual Supervision

Variational methods for Conditional Multimodal Deep Learning

Multimodal Adversarially Learned Inference with Factorized Discriminators

Hybrid VAE: Improving Deep Generative Models using Partial Observations