Langevin dynamics for the probability of finite state Markov processes

Wuchen Li

2024-09-19

Abstract:We study gradient drift-diffusion processes on a probability simplex set with finite state Wasserstein metrics, namely finite state Wasserstein common noises. A fact is that the Kolmogorov transition equation of finite reversible Markov processes satisfies the gradient flow of entropy in finite state Wasserstein space. This paper proposes to perturb finite state Markov processes with Wasserstein common noises. In this way, we introduce a class of stochastic reversible Markov processes. We also define stochastic transition rate matrices, namely Wasserstein Q-matrices, for the proposed stochastic Markov processes. We then derive the functional Fokker-Planck equation in the probability simplex, whose stationary distribution is a Gibbs distribution of entropy functional in a simplex set. Several examples of Wasserstein drift-diffusion processes on a two-point state space are presented.

Probability,Optimization and Control

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to define and study the drift - diffusion process of Markov processes with Wasserstein common noise under the Wasserstein metric in a finite - state space. Specifically, the paper proposes a method of perturbing a finite - state reversible Markov process by adding Wasserstein common noise on the finite - state probability simplex set, thereby introducing a new class of stochastically reversible Markov processes. These processes are characterized in that their transition rate matrices (i.e., Wasserstein Q - matrices) contain not only the classical transition rates but also Brownian motion coefficients that depend on the probability density.

### Main contributions of the paper

1. **Define Wasserstein Q - matrix**:
- Propose the concept of Wasserstein Q - matrix, which is used to describe the transition rates of Markov processes with Wasserstein common noise in a finite - state space.
- This matrix includes not only the classical transition rates but also a geometric diffusion term derived from the Wasserstein metric.

2. **Derive the functional Fokker - Planck equation**:
- Derive the functional Fokker - Planck equation of Markov processes with Wasserstein common noise in a finite - state Wasserstein space.
- Prove that the stationary solution of this equation is a Gibbs distribution, which has the form \( P^*(p)=\frac{1}{Z}e^{-\frac{1}{\beta}V(p)}\Pi(p)^{-\frac{1}{2}}\), where \( Z \) is a normalization constant, \( V(p) \) is a potential function, and \( \Pi(p) \) is the product of the eigenvalues of the Laplacian matrix.

3. **Numerical and analytical examples**:
- Through specific numerical simulations and analyses, show the behavior of the Wasserstein drift - diffusion process in a two - point state space.
- For example, when using the geometric mean as an activation function, give the specific form of the SDE and perform numerical simulations to show the characteristics of the trajectory and the stationary density.

### Formula summary

- **Wasserstein Q - matrix**:
\[
Q_{W_{ij}}(p,\beta,\dot{B})=\begin{cases}
Q_{ij}+a_{ij}(p)&\text{if }j\neq i\\
-\sum_{k\in N(i)}(Q_{ik}+a_{ik}(p))&\text{if }j = i
\end{cases}
\]
where,
\[
a_{ij}(p)=\frac{1}{p_i}\max\{0,A_{ji}(p)\}
\]
\[
A_{ij}(p)=\beta\omega_{ij}\theta_{ij}(p)\left(\frac{\partial}{\partial p_j}-\frac{\partial}{\partial p_i}\right)\log\Pi(p)^{\frac{1}{2}}\theta_{ij}(p)+\sqrt{\beta\omega_{ij}\theta_{ij}(p)}(\dot{B}_{ij}(t)-\dot{B}_{ji}(t))
\]

- **Functional Fokker - Planck equation**:
\[
\frac{\partial P(t,p)}{\partial t}+\nabla_p\cdot\left(P(t,p)\sum_{j = 1}^n[Q_{ji}p_j-Q_{ij}p_i]\right)=\beta L_{W}^*P(t,p)
\]