Abstract:As a computational alternative to Markov chain Monte Carlo approaches, variational inference (VI) is becoming more and more popular for approximating intractable posterior distributions in large-scale Bayesian models due to its comparable efficacy and superior efficiency. Several recent works provide theoretical justifications of VI by proving its statistical optimality for parameter estimation under various settings; meanwhile, formal analysis on the algorithmic convergence aspects of VI is still largely lacking. In this paper, we consider the common coordinate ascent variational inference (CAVI) algorithm for implementing the mean-field (MF) VI towards optimizing a Kullback--Leibler divergence objective functional over the space of all factorized distributions. Focusing on the two-block case, we analyze the convergence of CAVI by leveraging the extensive toolbox from functional analysis and optimization. We provide general conditions for certifying global or local exponential convergence of CAVI. Specifically, a new notion of generalized correlation for characterizing the interaction between the constituting blocks in influencing the VI objective functional is introduced, which according to the theory, quantifies the algorithmic contraction rate of two-block CAVI. As illustrations, we apply the developed theory to a number of examples, and derive explicit problem-dependent upper bounds on the algorithmic contraction rate.

On Convergence of Binary Trust-Region Steepest Descent

A Trust Region-Type Method for Solving Monotone Variational Inequality

Convergence and worst-case complexity of adaptive Riemannian trust-region methods for optimization on manifolds

Globally Convergent Composite-Step Trust-Region Framework for Model-Based Real-Time Optimization

On convergence of the generalized Lanczos trust-region method for trust-region subproblems

A TRUST-REGION METHOD FOR NONSMOOTH NONCONVEX OPTIMIZATION

The Convergence of the Generalized Lanczos Trust-Region Method for the Trust-Region Subproblem

Convergence depth control for interior point methods

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Beyond Nonconvexity: A Universal Trust-Region Method with New Analyses

Some convergently three-term trust region conjugate gradient algorithms under gradient function non-Lipschitz continuity

Fast convergence of trust-regions for non-isolated minima via analysis of CG on indefinite matrices

Local convergence analysis of an inexact trust-region method for nonsmooth optimization

On the Convergence of Coordinate Ascent Variational Inference

A simple and practical adaptive trust-region method

Stochastic Optimization for Nonconvex Problem with Inexact Hessian Matrix, Gradient, and Function

A consistently adaptive trust-region method

A nonmonotone trust region method for unconstrained optimization

An investigation of stochastic trust-region based algorithms for finite-sum minimization

An interior-point trust-region method for nonsmooth regularized bound-constrained optimization

A Convergence Analysis on the Iterative Trace Ratio Algorithm and Its Refinements