Abstract:Human perception is inherently multimodal. We integrate, for instance, visual, proprioceptive and tactile information into one experience. Hence, multimodal learning is of importance for building robotic systems that aim at robustly interacting with the real world. One potential model that has been proposed for multimodal integration is the multimodal variational autoencoder. A variational autoencoder (VAE) consists of two networks, an encoder that maps the data to a stochastic latent space and a decoder that reconstruct this data from an element of this latent space. The multimodal VAE integrates inputs from different modalities at two points in time in the latent space and can thereby be used as a controller for a robotic agent. Here we use this architecture and introduce information-theoretic measures in order to analyze how important the integration of the different modalities are for the reconstruction of the input data. Therefore we calculate two different types of measures, the first type is called single modality error and assesses how important the information from a single modality is for the reconstruction of this modality or all modalities. Secondly, the measures named loss of precision calculate the impact that missing information from only one modality has on the reconstruction of this modality or the whole vector. The VAE is trained via the evidence lower bound, which can be written as a sum of two different terms, namely the reconstruction and the latent loss. The impact of the latent loss can be weighted via an additional variable, which has been introduced to combat posterior collapse. Here we train networks with four different weighting schedules and analyze them with respect to their capabilities for multimodal integration.

An Information Theoretic Approach to the Autoencoder

Information Potential Auto-Encoders.

On the Information Plane of Autoencoders

Learning deep representations by mutual information estimation and maximization

A New Modal Autoencoder for Functionally Independent Feature Extraction

MAXIMUM ENTROPY AND MINIMAL MUTUAL INFORMATION IN A NONLINEAR MODEL

Knowledge-integrated autoencoder model

AE2-Nets: Autoencoder in Autoencoder Networks

Variational Encoders and Autoencoders : Information-theoretic Inference and Closed-form Solutions

Capacity-Approaching Autoencoders for Communications

Variational Lossy Autoencoder

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

Uncertainty Autoencoders: Learning Compressed Representations via Variational Information Maximization

Deep clustering by maximizing mutual information in variational auto-encoder

Binary autoencoder with random binary weights

InfoVAEGAN : learning joint interpretable representations by information maximization and maximum likelihood

Analyzing Multimodal Integration in the Variational Autoencoder from an Information-Theoretic Perspective

Autoencoder in Autoencoder Networks

TURBO: The Swiss Knife of Auto-Encoders

Information Theoretic-Learning auto-encoder