Abstract:We propose Tree Variational Autoencoder (TreeVAE), a new generative hierarchical clustering model that learns a flexible tree-based posterior distribution over latent variables. TreeVAE hierarchically divides samples according to their intrinsic characteristics, shedding light on hidden structures in the data. It adapts its architecture to discover the optimal tree for encoding dependencies between latent variables. The proposed tree-based generative architecture enables lightweight conditional inference and improves generative performance by utilizing specialized leaf decoders. We show that TreeVAE uncovers underlying clusters in the data and finds meaningful hierarchical relations between the different groups on a variety of datasets, including real-world imaging data. We present empirically that TreeVAE provides a more competitive log-likelihood lower bound than the sequential counterparts. Finally, due to its generative nature, TreeVAE is able to generate new samples from the discovered clusters via conditional sampling.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to discover hierarchical structures and hidden clusters in data and improve the performance of generative models. Specifically, the authors propose Tree Variational Autoencoder (TreeVAE), a new generative model that achieves hierarchical clustering by learning the flexible tree - shaped posterior distribution among latent variables. The main objectives of TreeVAE include: 1. **Discovering hidden structures in data**: TreeVAE aims to reveal the hidden clusters in data and their hierarchical relationships. By adaptively adjusting its architecture, TreeVAE can discover the optimal tree - shaped structure to encode the dependencies among latent variables. 2. **Improving generative performance**: TreeVAE uses specialized leaf decoders for conditional inference, thereby improving generative performance. Compared with traditional sequential methods, TreeVAE provides a more competitive log - likelihood lower bound. 3. **Light - weight conditional inference**: The tree - shaped generative architecture of TreeVAE makes conditional inference lightweight and can also generate new samples through conditional sampling. ### Specific problems - **How to effectively model the hierarchical dependencies among latent variables?** - TreeVAE represents the hierarchical relationships among latent variables by constructing a binary - tree structure. Each node and split is encoded by a neural network, thereby capturing the intrinsic characteristics of the data. - **How to optimize the performance of generative models?** - TreeVAE improves the performance of generative models by introducing the posterior distribution of the tree - shaped structure, especially performing better when dealing with complex data. - **How to discover meaningful hierarchical structures in an unsupervised manner?** - TreeVAE can automatically discover the hierarchical structures in data without relying on labels and organize them into interpretable tree - shaped structures. ### Paper contributions 1. **Proposing a new deep - probability method**: Combining the advantages of hierarchical clustering and deep generative models, learning the optimal generative binary tree to imitate the hierarchical structures existing in data. 2. **Extensive empirical evaluation**: Demonstrating the superior performance of TreeVAE in deep - level clustering on multiple datasets (such as MNIST, Fashion - MNIST, 20Newsgroups, and Omniglot). 3. **Introducing contrast - learning extension**: By integrating contrast - learning into the tree - shaped structure of TreeVAE, allowing the injection of domain knowledge or specific constraints, thereby achieving more accurate and context - related clustering. ### Summary of mathematical formulas - **Probability definition of the generative process**: \[ p_\theta(z_{P_l}, P_l) = p(z_0) \prod_{i \in P_l \setminus \{0\}} p(c_{pa(i) \to i} | z_{pa(i)}) p(z_i | z_{pa(i)}) \] - **Reconstruction loss**: \[ L_{rec} = \mathbb{E}_{q(z_{P_l}, P_l | x)} [\log p(x | z_{P_l}, P_l)] \] - **Kullback - Leibler divergence**: \[ KL(q(z_{P_l}, P_l | x) \| p(z_{P_l}, P_l)) = KL_{root} + KL_{nodes} + KL_{decisions} \] These formulas describe how TreeVAE learns the distributions of latent variables and paths through variational inference and how to calculate the evidence lower bound (ELBO) of the generative model.

Tree Variational Autoencoders

Hidden Talents of the Variational Autoencoder

Structured Generations: Using Hierarchical Clusters to guide Diffusion Models

Variational Tree Synthesis.

Conditional out-of-sample generation for unpaired data using trVAE

TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation

D-VAE: A Variational Autoencoder for Directed Acyclic Graphs

SGVAE: Sequential Graph Variational Autoencoder

Ladder Variational Autoencoders

Tree-based variational inference for Poisson log-normal models

Neighbor Embedding Variational Autoencoder

Recurrent Variational Autoencoders for Learning Nonlinear Generative Models in the Presence of Outliers.

scTree: Discovering Cellular Hierarchies in the Presence of Batch Effects in scRNA-seq Data

Three Variations on Variational Autoencoders

A Variational Approach to Bayesian Phylogenetic Inference

A Hierarchical Latent Structure for Variational Conversation Modeling

Generated Loss, Augmented Training, and Multiscale VAE

Green Generative Modeling: Recycling Dirty Data Using Recurrent Variational Autoencoders

A Contrastive Variational Graph Auto-Encoder for Node Clustering

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

Multimodal hierarchical Variational AutoEncoders with Factor Analysis latent space