Abstract:Neural operators have emerged as a powerful tool for learning the mapping between infinite-dimensional parameter and solution spaces of partial differential equations (PDEs). In this work, we focus on multiscale PDEs that have important applications such as reservoir modeling and turbulence prediction. We demonstrate that for such PDEs, the spectral bias towards low-frequency components presents a significant challenge for existing neural operators. To address this challenge, we propose a hierarchical attention neural operator (HANO) inspired by the hierarchical matrix approach. HANO features a scale-adaptive interaction range and self-attentions over a hierarchy of levels, enabling nested feature computation with controllable linear cost and encoding/decoding of multiscale solution space. We also incorporate an empirical $H^1$ loss function to enhance the learning of high-frequency components. Our numerical experiments demonstrate that HANO outperforms state-of-the-art (SOTA) methods for representative multiscale problems.

What problem does this paper attempt to address?

This paper focuses on addressing the issues encountered in neural operator learning for solving multiscale partial differential equations (MsPDEs). Existing neural operator methods suffer from "spectral bias" problem, where they tend to prioritize learning low-frequency components and struggle to capture the high-frequency characteristics in multiscale problems. This bias limits the model's ability to accurately represent fine details. To tackle this problem, the paper proposes a Hierarchical Attention Neural Operator (HANO) inspired by the hierarchical matrix approach. HANO features an adaptive range of interaction and a multilevel self-attention mechanism, which enables nested feature computations at controlled linear computational cost and encodes/decodes the multiscale solution space. Additionally, the paper introduces an empirical H1 loss function to further reduce spectral bias and enhance the learning of high-frequency components. Through numerical experiments, HANO demonstrates superior performance compared to the state-of-the-art methods on representative multiscale problems. These experiments include handling the Navier-Stokes equation (turbulent state) and high-wavenumber Helmholtz equation, showcasing the advantages of HANO in capturing multiscale characteristics. In summary, this paper aims to mitigate spectral bias in solving multiscale PDEs using HANO, improving the accuracy of data-driven forward and inverse solving, particularly when dealing with stochastic or parameterized coefficients.

Mitigating spectral bias for the multiscale operator learning