Understanding the impact of numerical solvers on inference for differential equation models

Richard Creswell,Katherine M. Shepherd,Ben Lambert,Gary R. Mirams,Chon Lok Lei,Simon Tavener,Martin Robinson,David J. Gavaghan

2023-07-03

Abstract:Most ordinary differential equation (ODE) models used to describe biological or physical systems must be solved approximately using numerical methods. Perniciously, even those solvers which seem sufficiently accurate for the forward problem, i.e., for obtaining an accurate simulation, may not be sufficiently accurate for the inverse problem, i.e., for inferring the model parameters from data. We show that for both fixed step and adaptive step ODE solvers, solving the forward problem with insufficient accuracy can distort likelihood surfaces, which may become jagged, causing inference algorithms to get stuck in local "phantom" optima. We demonstrate that biases in inference arising from numerical approximation of ODEs are potentially most severe in systems involving low noise and rapid nonlinear dynamics. We reanalyze an ODE changepoint model previously fit to the COVID-19 outbreak in Germany and show the effect of the step size on simulation and inference results. We then fit a more complicated rainfall-runoff model to hydrological data and illustrate the importance of tuning solver tolerances to avoid distorted likelihood surfaces. Our results indicate that when performing inference for ODE model parameters, adaptive step size solver tolerances must be set cautiously and likelihood surfaces should be inspected for characteristic signs of numerical issues.

Statistics Theory,Numerical Analysis

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper explores the impact of numerical solvers on the inference of ordinary differential equation (ODE) models. Specifically, the author is concerned with the problem that numerical solvers that perform well in the forward problem (i.e., obtaining accurate simulation results) may not be accurate enough in the inverse problem (i.e., inferring model parameters from data). The main research contents include:

1. **The impact of numerical errors on the calculation of the likelihood function**:
- Even if seemingly accurate enough numerical solvers are used in the forward problem, these solvers may lead to significant deviations in the inverse problem. Through theoretical analysis and experimental verification, the author shows how numerical errors distort the likelihood function surface, causing the inference algorithm to fall into local "phantom" optimal values.

2. **The impact of different numerical solvers**:
- The performance of fixed - step - size and adaptive - step - size ODE solvers in parameter inference is studied. The author finds that the fixed - step - size solver may lead to an overall shift of the likelihood function surface, while the adaptive - step - size solver may lead to jagged and discontinuous fluctuations on the surface.

3. **The impact in practical applications**:
- By re - analyzing an ODE model used for the spread of COVID - 19 in Germany and a rainfall - runoff model used in hydrology, the impact of step - size selection on simulation and inference results is demonstrated. The author emphasizes that when performing parameter inference, the tolerance of the adaptive - step - size solver must be set carefully, and the likelihood function surface should be checked for characteristic signs of numerical problems.

### Key conclusions

- **The severity of numerical errors**:
- Numerical errors are particularly severe in low - noise and fast - nonlinear - dynamics systems and may lead to significant inference deviations.

- **The difference between fixed - step - size and adaptive - step - size solvers**:
- The fixed - step - size solver may lead to an overall shift of the likelihood function surface, while the adaptive - step - size solver may lead to jagged and discontinuous fluctuations on the surface.

- **Recommendations in practical applications**:
- When performing parameter inference for ODE models, the tolerance of the adaptive - step - size solver should be set carefully, and the likelihood function surface should be checked for characteristic signs of numerical problems.

### Formula summary

- **Likelihood function**:
\[
\log p(y_1, \ldots, y_N \mid \theta, \sigma)=-\frac{N}{2} \log(2\pi)-\frac{N}{2} \log(\sigma^2)-\frac{1}{2\sigma^2} \sum_{i = 1}^N (y_i - g(x(t_i; \theta)))^2
\]

- **The impact of numerical errors on the likelihood function**:
\[
|L - L'|\leq \sum_{i = 1}^N \left(\frac{K^2}{2\sigma^2} |e(t_i)|^2+\frac{K}{\sigma^2} |e(t_i)| |y_i - g(x(t_i; \theta))|\right)
\]
where \(e(t_i)=x(t_i; \theta)-\hat{x}_i\) is the global truncation error, and \(K\) is the Lipschitz constant of the observation function \(g\).

Through these studies, the author hopes to raise researchers' awareness of the potential impact of numerical solvers in parameter inference and provide practical guidance and suggestions.