Abstract:We propose Scalable Mechanistic Neural Network (S-MNN), an enhanced neural network framework designed for scientific machine learning applications involving long temporal sequences. By reformulating the original Mechanistic Neural Network (MNN) (Pervez et al., 2024), we reduce the computational time and space complexities from cubic and quadratic with respect to the sequence length, respectively, to linear. This significant improvement enables efficient modeling of long-term dynamics without sacrificing accuracy or interpretability. Extensive experiments demonstrate that S-MNN matches the original MNN in precision while substantially reducing computational resources. Consequently, S-MNN can drop-in replace the original MNN in applications, providing a practical and efficient tool for integrating mechanistic bottlenecks into neural network models of complex dynamical systems.

What problem does this paper attempt to address?

The main problem this paper attempts to address is improving the computational efficiency and memory usage of Mechanistic Neural Networks (MNN) when handling long time series data. Specifically, the paper proposes a Scalable Mechanistic Neural Network (S-MNN) that redesigns the basic linear system of MNN, reducing the computational time and space complexity from cubic and quadratic levels with respect to sequence length to linear levels. This improvement allows S-MNN to efficiently model long-term dynamic processes without sacrificing accuracy or interpretability, making it suitable for applications involving long time spans or high-resolution time data, such as climate records. ### Main Contributions: 1. **Complexity Reduction**: By eliminating slack variables and central difference constraints, the quadratic programming problem is simplified to a least squares regression, giving the left-side matrix a banded structure. This allows the use of efficient algorithms, reducing the computational time and space complexity to linear levels with respect to sequence length. 2. **Efficient Solver Design**: An efficient solver is developed that leverages the inherent sparsity and banded structure of the linear system, optimizing GPU execution and fully utilizing parallel computing capabilities, significantly increasing speed. 3. **Long-term Sequence Modeling**: The effectiveness of S-MNN is validated through multiple benchmarks, including the discovery of control equations for the Lorenz system, solving the Korteweg-de Vries (KdV) partial differential equation, and long-term sea surface temperature (SST) prediction. Experimental results show that S-MNN significantly reduces computational time and memory usage while maintaining the same accuracy as the original MNN. ### Problems Addressed: - **Computational Efficiency**: The original MNN's computational time and memory usage increase sharply with sequence length when handling long time series data, limiting its practical application. S-MNN significantly reduces computational complexity through optimized algorithms and solver design, enabling it to handle longer time series. - **Memory Usage**: The original MNN consumes excessive memory when processing large-scale data, making it unable to run on existing hardware. S-MNN reduces memory requirements, allowing the model to run efficiently on existing hardware. - **Practical Applications**: S-MNN demonstrates its advantages in practical application scenarios such as climate data, capable of handling long time-span data and providing a practical tool for scientific research. In summary, this paper addresses the computational efficiency and memory usage issues of the original MNN when handling long time series data by proposing S-MNN, providing a more efficient and practical solution for modeling complex dynamic systems in the field of scientific machine learning.

Scalable Mechanistic Neural Networks

Mechanistic Neural Networks for Scientific Machine Learning

Constructing Lightweight and Efficient Spiking Neural Networks for EEG-based Motor Imagery Classification

A New Neural Network Model And Its Application

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

HybridSNN: Combining Bio-Machine Strengths by Boosting Adaptive Spiking Neural Networks.

MAP-SNN: Mapping spike activities with multiplicity, adaptability, and plasticity into bio-plausible spiking neural networks

CSNN: an Augmented Spiking Based Framework with Perceptron-Inception

Standard neural network model and its application

Structured and Balanced Multi-component and Multi-layer Neural Networks

Learning Long Sequences in Spiking Neural Networks

Advancing Spiking Neural Networks towards Multiscale Spatiotemporal Interaction Learning

Training all-mechanical neural networks for task learning through in situ backpropagation

SPikE-SSM: A Sparse, Precise, and Efficient Spiking State Space Model for Long Sequences Learning

Obtaining Optimal Spiking Neural Network in Sequence Learning via CRNN-SNN Conversion

Mechanical Neural Networks with Explicit and Robust Neurons

A universal ANN-to-SNN framework for achieving high accuracy and low latency deep Spiking Neural Networks

BrainTransformers: SNN-LLM

SIT: A Bionic and Non-Linear Neuron for Spiking Neural Network