Abstract:A new particle-based sampling and approximate inference method, based on electrostatics and Newton mechanics principles, is introduced with theoretical ground, algorithm design and experimental validation. This method simulates an interacting particle system (IPS) where particles, i.e. the freely-moving negative charges and spatially-fixed positive charges with magnitudes proportional to the target distribution, interact with each other via attraction and repulsion induced by the resulting electric fields described by Poisson's equation. The IPS evolves towards a steady-state where the distribution of negative charges conforms to the target distribution. This physics-inspired method offers deterministic, gradient-free sampling and inference, achieving comparable performance as other particle-based and MCMC methods in benchmark tasks of inferring complex densities, Bayesian logistic regression and dynamical system identification. A discrete-time, discrete-space algorithmic design, readily extendable to continuous time and space, is provided for usage in more general inference problems occurring in probabilistic machine learning scenarios such as Bayesian inference, generative modelling, and beyond.
What problem does this paper attempt to address?
This paper proposes a new particle sampling and approximate inference method based on electrostatics and Newtonian mechanics. Probability methods, such as Bayesian inference and generative modeling, are widely used in many learning and decision tasks, which often involve estimating the distribution of the quantity of interest. However, when the target density is not easily accessible, such as high-dimensional complex integration or unavailable likelihood function, approximate inference methods such as Markov Chain Monte Carlo (MCMC) and Variational Inference (VI) need to be utilized.
The paper introduces a method for simulating an Interacting Particle System (IPS) in which freely moving negative charges interact with fixed positive charges in space, and the magnitudes of the positive charges are proportional to the target distribution. The interactions between these charges, with attractive and repulsive forces, gradually align the negative charge distribution with the target distribution. This method provides deterministic, gradient-free sampling and inference, and experiments demonstrate its performance on benchmark tasks such as complex density estimation, Bayesian logistic regression, and dynamical system identification, which is comparable to other particle-based and MCMC methods.
The limitations of existing methods are also discussed in the paper, such as the computational intensity and slow convergence of MCMC, as well as the optimization challenges and bias of variational inference. Particle-based variational inference methods combine the advantages of sampling and variational inference, being fast, accurate, and deterministic, but most methods still require gradient information.
The main contribution of this paper is the introduction of a physics-based particle-based variational inference method called EParVI, which simulates a physics-inspired IPS for sampling and approximate inference. This method is deterministic, gradient-free, and attractive in terms of flexibility, expressive power, and accuracy. It only requires queries to the target density without its gradient information. In this way, EParVI can handle multi-modal and sequential inference problems, providing a good trade-off between time and accuracy.
The organization of the paper includes method description, experimental validation, discussion, conclusion, and future work. The authors also provide detailed theoretical background, algorithm design, and key results.