Abstract:The cumulative distribution network (CDN) is a recently developed class of probabilistic graphical models (PGMs) permitting a copula factorization, in which the CDF, rather than the density, is factored. Despite there being much recent interest within the machine learning community about copula representations, there has been scarce research into the CDN, its amalgamation with copula theory, and no evaluation of its performance. Algorithms for inference, sampling, and learning in these models are underdeveloped compared those of other PGMs, hindering widerspread use. One advantage of the CDN is that it allows the factors to be parameterized as copulae, combining the benefits of graphical models with those of copula theory. In brief, the use of a copula parameterization enables greater modelling flexibility by separating representation of the marginals from the dependence structure, permitting more efficient and robust learning. Another advantage is that the CDN permits the representation of implicit latent variables, whose parameterization and connectivity are not required to be specified. Unfortunately, that the model can encode only latent relationships between variables severely limits its utility. In this thesis, we present inference, learning, and sampling for CDNs, and further the state-of-the-art. First, we explain the basics of copula theory and the representation of copula CDNs. Then, we discuss inference in the models, and develop the first sampling algorithm. We explain standard learning methods, propose an algorithm for learning from data missing completely at random (MCAR), and develop a novel algorithm for learning models of arbitrary treewidth and size. Properties of the models and algorithms are investigated through Monte Carlo simulations. We conclude with further discussion of the advantages and limitations of CDNs, and suggest future work.

Why (and When and How) Contrastive Divergence Works

Convergence of Contrastive Divergence Algorithm in Exponential Family

Adiabatic Persistent Contrastive Divergence Learning

A Neighbourhood-Based Stopping Criterion for Contrastive Divergence Learning

Training Energy-Based Models with Diffusion Contrastive Divergences

On the connection between Noise-Contrastive Estimation and Contrastive Divergence

Inference, Sampling, and Learning in Copula Cumulative Distribution Networks

Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning

Computing Marginal and Conditional Divergences between Decomposable Models with Applications

A Cyclic Contrastive Divergence Learning Algorithm for High-Order RBMs

Statistical divergences in high-dimensional hypothesis testing and a modern technique for estimating them

Convergence of Unadjusted Langevin in High Dimensions: Delocalization of Bias

The Representation Jensen-Shannon Divergence

On the Convergence of Coordinate Ascent Variational Inference

Bayesian Experimental Design via Contrastive Diffusions

Understanding Contrastive Learning via Distributionally Robust Optimization

Rates of convergence and normal approximations for estimators of local dependence random graph models

Where to Diffuse, How to Diffuse, and How to Get Back: Automated Learning for Multivariate Diffusions

Theoretical and Computational Guarantees of Mean Field Variational Inference for Community Detection

Parameter Estimation Procedures for Exponential-Family Random Graph Models on Count-Valued Networks: A Comparative Simulation Study

Can RBMs be trained with zero step contrastive divergence?