Abstract:We propose a parallel (distributed) version of the spectral proper orthogonal decomposition (SPOD) technique. The parallel SPOD algorithm distributes the spatial dimension of the dataset preserving time. This approach is adopted to preserve the non-distributed fast Fourier transform of the data in time, thereby avoiding the associated bottlenecks. The parallel SPOD algorithm is implemented in the PySPOD library and makes use of the standard message passing interface (MPI) library, implemented in Python via mpi4py . An extensive performance evaluation of the parallel package is provided, including strong and weak scalability analyses. The open-source library allows the analysis of large datasets of interest across the scientific community. Here, we present applications in fluid dynamics and geophysics, that are extremely difficult (if not impossible) to achieve without a parallel algorithm. This work opens the path toward modal analyses of big quasi-stationary data, helping to uncover new unexplored spatio-temporal patterns. Program summary Program Title: PySPOD CPC Library link to program files: https://doi.org/10.17632/jf5bf26jcj.1 Developer's repository link: https://github.com/MathEXLab/PySPOD Licensing provisions: MIT License Programming language: Python Nature of problem: Large spatio-temporal datasets may contain coherent patterns that can be leveraged to better understand, model, and possibly predict the behaviour of complex dynamical systems. To this end, modal decomposition methods, such as the proper orthogonal decomposition (POD) and its spectral counterpart (SPOD), constitute powerful tools. The SPOD algorithm allows the systematic identification of space-time coherent patterns. This can be used to understand better the physics of the process of interest, and provide a path for mathematical modeling, including reduced order modeling. The SPOD algorithm has been successfully applied to fluid dynamics, geophysics and other domains. However, the existing open-source implementations are serial, and they prevent running on the increasingly large datasets that are becoming available, especially in computational physics. The inability to analyse via SPOD large dataset in turn prevents unlocking novel mechanisms and dynamical behaviours in complex systems. Solution method: We provide an open-source parallel (MPI distributed) code, namely PySPOD, that is able to run on large datasets (the ones considered in the present paper reach about 200 Terabytes). The code is built on the previous serial open-source code PySPOD that was published in https://joss.theoj.org/papers/10.21105/joss.02862.pdf . The new parallel implementation is able to scale on several nodes (we show both weak and strong scalability) and solve some of the bottlenecks that are commonly found at the I/O stage. The current parallel code allows running on datasets that was not easy or possible to analyse with serial SPOD algorithms, hence providing a path towards unlocking novel findings in computational physics. Additional comments including restrictions and unusual features: The code comes with a set of built-in postprocessing tools, for visualizing the results. It also comes with extensive continuous integration, documentation, and tutorials, as well as a dedicated website in addition to the associated GiHub repository. Within the package we also provide a parallel implementation of the proper orthogonal decomposition (POD), that leverages the I/O parallel capabilities of the SPOD algorithm.

PyfastSPM: A Python package to convert 1D FastSPM data streams into publication quality movies

The new FAST module: a portable and transparent add-on module for time-resolved investigations with commercial scanning probe microscopes

Wavefront Modulation with Spatial Light Modulator for Fourier Ptychographic Microscopy

A Fast Algorithm for Scanning Transmission Electron Microscopy (STEM) Imaging and 4D-STEM Diffraction Simulations

Improved accuracy and speed in scanning probe microscopy by image reconstruction from non-gridded position sensor data

Micro Fourier Transform Profilometry (μftp): 3D Shape Measurement at 10,000 Frames Per Second

Micro Fourier Transform Profilometry ($μ$FTP): 3D shape measurement at 10,000 frames per second

PySTACHIO: Python Single-molecule TrAcking stoiCHiometry Intensity and simulatiOn, a flexible, extensible, beginner-friendly and optimized program for analysis of single-molecule microscopy

An open-source software package for on-the-fly deskewing and live viewing of volumetric lightsheet microscopy data

A streaming multi-GPU implementation of image simulation algorithms for scanning transmission electron microscopy

High Speed Piezoresponse Force Microscopy: <1 Frame Per Second Nanoscale Imaging

The Scatman: an approximate method for fast wide-angle scattering simulations

Quantitative mapping of smooth topographic landscapes produced by thermal scanning-probe lithography

Micro Fourier Transform Profilometry (μFTP): 3D imaging at 10,000 fps

Particle Smoother to Assimilate Asynchronous Movie Data of High-Speed AFM with MD Simulations

A customizable software for fast reduction and analysis of large X-ray scattering data sets: applications of the new DPDAK package to small-angle X-ray scattering and grazing-incidence small-angle X-ray scattering

Unlocking massively parallel spectral proper orthogonal decompositions in the PySPOD package

PySPH: A Python-based Framework for Smoothed Particle Hydrodynamics

A Fast Fourier Ptychographic Microscope Method with Biomedical Application

Advancing Scanning Probe Microscopy Simulations: A Decade of Development in Probe-Particle Models

Integration of Scanning Probe Microscope with High-Performance Computing: fixed-policy and reward-driven workflows implementation