Hierarchical Neural Operator Transformer with Learnable Frequency-aware Loss Prior for Arbitrary-scale Super-resolution

Xihaier Luo,Xiaoning Qian,Byung-Jun Yoon

2024-05-21

Abstract:In this work, we present an arbitrary-scale super-resolution (SR) method to enhance the resolution of scientific data, which often involves complex challenges such as continuity, multi-scale physics, and the intricacies of high-frequency signals. Grounded in operator learning, the proposed method is resolution-invariant. The core of our model is a hierarchical neural operator that leverages a Galerkin-type self-attention mechanism, enabling efficient learning of mappings between function spaces. Sinc filters are used to facilitate the information transfer across different levels in the hierarchy, thereby ensuring representation equivalence in the proposed neural operator. Additionally, we introduce a learnable prior structure that is derived from the spectral resizing of the input data. This loss prior is model-agnostic and is designed to dynamically adjust the weighting of pixel contributions, thereby balancing gradients effectively across the model. We conduct extensive experiments on diverse datasets from different domains and demonstrate consistent improvements compared to strong baselines, which consist of various state-of-the-art SR methods.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The problem addressed in the paper is arbitrary-scale super-resolution (ASR) of scientific data. Scientific data often involves complex challenges such as continuity, multiscale physical phenomena, and fine structures of high-frequency signals. Existing deep learning baseline methods are often limited to fixed-scale enhancement when dealing with these data, requiring separate training of models for each scaling factor, which restricts their practicality.

To tackle this problem, the paper proposes a Hierarchical Neural Operator Transformer that utilizes a Galerkin-type self-attention mechanism to efficiently learn mappings between function spaces. The core of this approach lies in the hierarchical design, promoting information transfer between different levels through sinc filters, ensuring equivalence of neural operator representations. In addition, the paper introduces a learnable frequency-aware loss prior, which is obtained by spectral resampling of the input data, dynamically adjusting pixel contribution weights during the training process to balance high-frequency and low-frequency regions in the model.

Through extensive experiments on diverse datasets from multiple domains, this method demonstrates consistent improvements over existing state-of-the-art super-resolution methods, proving its effectiveness in handling arbitrary-scale super-resolution tasks.