Abstract:Recently there has been great interest in operator learning, where networks learn operators between function spaces from an essentially infinite-dimensional perspective. In this work we present results for when the operators learned by these networks are injective and surjective. As a warmup, we combine prior work in both the finite-dimensional ReLU and operator learning setting by giving sharp conditions under which ReLU layers with linear neural operators are injective. We then consider the case the case when the activation function is pointwise bijective and obtain sufficient conditions for the layer to be injective. We remark that this question, while trivial in the finite-rank case, is subtler in the infinite-rank case and is proved using tools from Fredholm theory. Next, we prove that our supplied injective neural operators are universal approximators and that their implementation, with finite-rank neural networks, are still injective. This ensures that injectivity is not `lost' in the transcription from analytical operators to their finite-rank implementation with networks. Finally, we conclude with an increase in abstraction and consider general conditions when subnetworks, which may be many layers deep, are injective and surjective and provide an exact inversion from a `linearization.' This section uses general arguments from Fredholm theory and Leray-Schauder degree theory for non-linear integral equations to analyze the mapping properties of neural operators in function spaces. These results apply to subnetworks formed from the layers considered in this work, under natural conditions. We believe that our work has applications in Bayesian UQ where injectivity enables likelihood estimation and in inverse problems where surjectivity and injectivity corresponds to existence and uniqueness, respectively.

The inverse problem for neural networks

Preservation of Local Linearity by Neighborhood Subspace Scaling for Solving the Pre-Image Problem.

Neumann Networks for Inverse Problems in Imaging

Inversion of Integral Models: a Neural Network Approach

Quadratic neural networks for solving inverse problems

Provable Preimage Under-Approximation for Neural Networks (Full Version)

Convergence Guarantees of Overparametrized Wide Deep Inverse Prior

Convergence and Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems

Regularization of Inverse Problems by Neural Networks

Convolutional Neural Networks Analyzed via Inverse Problem Theory and Sparse Representations

Understanding Neural Network Systems for Image Analysis using Vector Spaces and Inverse Maps

Neural Networks, Hypersurfaces, and Radon Transforms

Neural network augmented inverse problems for PDEs

Globally injective and bijective neural operators

Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems

To be or not to be stable, that is the question: understanding neural networks for inverse problems

Best Approximation and Inverse Results for Neural Network Operators

Untrained Neural Network Priors for Inverse Imaging Problems: A Survey

Big in Japan: Regularizing networks for solving inverse problems