Abstract:Multispectral and Hyperspectral Image Fusion (MHIF) is a task that aims to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI) acquired on the same scene to obtain a high-resolution hyperspectral image (HR-HSI). Benefiting from the powerful inductive bias capability, convolutional neural network (CNN) based methods have achieved great success for the MHIF task. However, they lack flexibility when processing multi-scale images and require convolution structures be stacked to enhance performance. Implicit neural representation (INR) has recently achieved good performance and interpretability in 2D processing tasks thanks to its ability to locally interpolate samples and utilize multimodal content, such as pixels and coordinates. Although INR-based approaches show promising results, they put additional demands on high-frequency information (e.g., positional encoding). In this paper, we propose the use of the HR-MSI as high-frequency detail auxiliary input, thus introducing a new INR-based hyperspectral fusion function called implicit neural feature fusion function (INF3). The method overcomes the inherent shortcomings of vanilla INR thereby solving the MHIF problem. Specifically, our INF3 designs a dual high-frequency fusion (DHFF) structure that obtains high-frequency information from HR-MSI and LR-HSI fusing them with coordinate information. Moreover, the proposed INF3 incorporates a parameter-free method called INR with cosine similarity (INR-CS) that uses cosine similarity to generate local weights through feature vectors. Relied upon INF3, we build an implicit neural fusion network (INFN) that achieves state-of-the-art performance for the MHIF task on two public datasets, i.e., CAVE and Harvard. It also reaches the advanced level on the Pansharpening task, proving the flexibility of the proposed approach.

FreqINR: Frequency Consistency for Implicit Neural Representation with Adaptive DCT Frequency Loss

Frequency-Relevant Residual Learning for Multi-Modal Image Denoising.

Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion

FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions

Synergizing frequency domain and texture-aware implicit module for MRI arbitrary-scale super-resolution

SS-INR: Spatial-Spectral Implicit Neural Representation Network for Hyperspectral and Multispectral Image Fusion

INF3: Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion

Lightweight Multiresolution Feature Fusion Network for Spectral Super-Resolution

FreqNet: A Frequency-domain Image Super-Resolution Network with Dicrete Cosine Transform

FINER++: Building a Family of Variable-periodic Functions for Activating Implicit Neural Representation

Revisiting Implicit Neural Representations in Low-Level Vision

Frequency Consistent Adaptation for Real World Super Resolution

Freq-Mip-AA : Frequency Mip Representation for Anti-Aliasing Neural Radiance Fields

Dynamic Implicit Image Function for Efficient Arbitrary-Scale Image Representation

FFEINR: Flow Feature-Enhanced Implicit Neural Representation for Spatio-temporal Super-Resolution

Recovering Realistic Details for Magnification-Arbitrary Image Super-Resolution

Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion

FFEINR: flow feature-enhanced implicit neural representation for spatiotemporal super-resolution

Disorder-invariant Implicit Neural Representation

Implicit Neural Representation Learning for Hyperspectral Image Super-Resolution

Implicit Neural Representations with Fourier Kolmogorov-Arnold Networks