Abstract:Recently there has been great interest in operator learning, where networks learn operators between function spaces from an essentially infinite-dimensional perspective. In this work we present results for when the operators learned by these networks are injective and surjective. As a warmup, we combine prior work in both the finite-dimensional ReLU and operator learning setting by giving sharp conditions under which ReLU layers with linear neural operators are injective. We then consider the case the case when the activation function is pointwise bijective and obtain sufficient conditions for the layer to be injective. We remark that this question, while trivial in the finite-rank case, is subtler in the infinite-rank case and is proved using tools from Fredholm theory. Next, we prove that our supplied injective neural operators are universal approximators and that their implementation, with finite-rank neural networks, are still injective. This ensures that injectivity is not `lost' in the transcription from analytical operators to their finite-rank implementation with networks. Finally, we conclude with an increase in abstraction and consider general conditions when subnetworks, which may be many layers deep, are injective and surjective and provide an exact inversion from a `linearization.' This section uses general arguments from Fredholm theory and Leray-Schauder degree theory for non-linear integral equations to analyze the mapping properties of neural operators in function spaces. These results apply to subnetworks formed from the layers considered in this work, under natural conditions. We believe that our work has applications in Bayesian UQ where injectivity enables likelihood estimation and in inverse problems where surjectivity and injectivity corresponds to existence and uniqueness, respectively.

Decomposition of one-layer neural networks via the infinite sum of reproducing kernel Banach spaces

A reproducing kernel approach to Lebesgue decomposition

Nonuniform sampling, reproducing kernels, and the associated Hilbert spaces

The $L^\infty$ Learnability of Reproducing Kernel Hilbert Spaces

Universality and Optimality of Structured Deep Kernel Networks

Training Neural Networks Using Reproducing Kernel Space Interpolation and Model Reduction

Kernel embedding of measures and low-rank approximation of integral operators

Adaptive indefinite kernels in hyperbolic spaces

Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regime

Neural reproducing kernel Banach spaces and representer theorems for deep networks

Analysis of Regularized Least-Squares in Reproducing Kernel Kreĭn Spaces

An algebra structure for reproducing kernel Hilbert spaces

From deep to Shallow: Equivalent Forms of Deep Networks in Reproducing Kernel Krein Space and Indefinite Support Vector Machines

Globally injective and bijective neural operators

Deep Learning with Kernels through RKHM and the Perron-Frobenius Operator

Which Spaces can be Embedded in $L_p$-type Reproducing Kernel Banach Space? A Characterization via Metric Entropy

Hypothesis Spaces for Deep Learning

On the Approximation Properties of Random ReLU Features

Approximation of RKHS Functionals by Neural Networks

Countable Tensor Products of Hermite Spaces and Spaces of Gaussian Kernels

Duality in RKHSs with Infinite Dimensional Outputs: Application to Robust Losses