Abstract:Abstract Several observation types (e.g., geostationary satellite and Doppler radar observations) have recently been found to exhibit strong spatial error correlations. Including these error statistics in data assimilation for numerical weather prediction can improve analysis quality and forecast skill. Moreover, it allows for increases in the spatial density of observations assimilated, which is needed for the provision of information on appropriate scales for high‐resolution forecasting. However, introducing correlated error statistics may increase the computational complexity and parallel communication costs of matrix‐vector products involving observation precision matrices (inverse observation error covariance matrices). Without new approaches, we cannot take full advantage of new observation uncertainty estimates. We develop a new numerical approximation method based on a particular type of fast multipole method and a domain localization approach. The basic idea is to divide the observation domain into boxes and then separate calculations of matrix‐vector products according to the partition. These calculations can be done in parallel with very low communication overheads. The new method is easy to implement and parallelize, and it is applicable to a wide variety of observation precision matrices. We applied the new method to a simple variational data assimilation problem and found that the computational cost of the variational minimization was dramatically reduced while preserving analysis accuracy across a range of scales. The new method has the potential to be used as an efficient technique for practical applications where a large number of observations with mutual error correlations need to be assimilated quickly.

Fast post-hoc method for updating moments of large datasets

Recalculate Without Recomputing

Online updating mode learning for streaming datasets

Online Updating of Statistical Inference in the Big Data Setting

Distributed Bayesian Posterior Sampling Via Moment Sharing

An information reuse-based method for reliability updating

Adaptive Change Point Monitoring for High-Dimensional Data

A Distribution-Free Method for Change Point Detection in Non-Sparse High Dimensional Data

On Simplifying Large-Scale Spatial Vectors: Fast, Memory-Efficient, and Cost-Predictable k-means

NEWMA: a new method for scalable model-free online change-point detection

An improved high-order statistical moment method for structural reliability analysis with insufficient data

A Novel Localized Fast Multipole Method for Computations With Spatially Correlated Observation Error Statistics in Data Assimilation

Post-Estimation Smoothing: A Simple Baseline for Learning with Side Information

Knowable Moments in Stochastics: Knowing Their Advantages

Asymptotic distribution-free change-point detection based on interpoint distances for high-dimensional data

Large-Dimensional Factor Analysis Without Moment Constraints

Moment-Based Quantile Sketches for Efficient High Cardinality Aggregation Queries

Direct Regression Modelling of High-Order Moments in Big Data

The Polynomial Stein Discrepancy for Assessing Moment Convergence

Fast and Low Memory Cost Matrix Factorization: Algorithm, Analysis, and Case Study.

Large-Sample Significance Levels from Multiply Imputed Data Using Moment-Based Statistics and an F-Reference Distribution