Abstract:We consider the problem of learning a linear operator $\theta$ between two Hilbert spaces from empirical observations, which we interpret as least squares regression in infinite dimensions. We show that this goal can be reformulated as an inverse problem for $\theta$ with the feature that its forward operator is generally non-compact (even if $\theta$ is assumed to be compact or of $p$-Schatten class). However, we prove that, in terms of spectral properties and regularisation theory, this inverse problem is equivalent to the known compact inverse problem associated with scalar response regression. Our framework allows for the elegant derivation of dimension-free rates for generic learning algorithms under Hölder-type source conditions. The proofs rely on the combination of techniques from kernel regression with recent results on concentration of measure for sub-exponential Hilbertian random variables. The obtained rates hold for a variety of practically-relevant scenarios in functional regression as well as nonlinear regression with operator-valued kernels and match those of classical kernel regression with scalar response.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the problem of learning a linear operator $ \theta $ between two Hilbert spaces. Specifically, it is to learn this linear operator from empirical observations, which can be interpreted as a least - squares regression problem in infinite dimensions. The core objectives of the paper are: 1. **Define the problem**: Consider square - integrable random variables $ X $ and $ Y $ taking values from two Hilbert spaces $ X $ and $ Y $. The goal is to minimize $ \mathbb{E}[\|Y - \theta X\|^2_Y] $, where $ \theta \in L(X, Y) $ is a bounded linear operator. If at least one of $ X $ or $ Y $ is infinite - dimensional, then $ L(X, Y) $ is also infinite - dimensional, so this is an infinite - dimensional regression problem. 2. **Reformulate as an inverse problem**: The paper reformulates this problem as an inverse problem, whose forward operator is usually non - compact, even if it is assumed that $ \theta $ is compact or belongs to the $ p $-Schatten class. However, the authors prove that in terms of spectral properties and regularization theory, this inverse problem is equivalent to the known compact inverse problem related to scalar - response regression. 3. **Regularization and convergence rates**: The paper provides a framework for deriving dimension - free convergence rates of general learning algorithms under Hölder - type source conditions. The proof relies on kernel regression techniques and recent results on concentration measures of sub - exponential Hilbert random variables. 4. **Practical applications**: The paper discusses the applications of this framework in practical scenarios such as functional linear regression, non - parametric regression with vector - valued kernels, conditional mean embedding, and Hilbert time - series inference. ### Main contributions - **Mathematical background**: Provides the mathematical background of infinite - dimensional linear regression, especially the treatment methods in the general context of inverse problems. - **Functional analysis framework**: Provides a functional analysis framework for the statistical analysis of regularized empirical solutions. - **Regularization theory**: Proves that even in non - compact cases, inverse problems can be solved by regularization methods, thereby solving the infinite - dimensional regression problem. - **Convergence rates**: Derives the basic convergence rates of general learning algorithms under Hölder - type source conditions, and these rates match those under classical kernel regression. ### Solution of specific problems - **Existence**: Discusses the necessary and sufficient conditions for the existence of the regression solution $ \theta^* $ and gives in which practical scenarios these conditions hold. - **Uniqueness**: Proves that when the distribution $ L(X) $ is non - degenerate, the regression solution $ \theta^* $ is unique. - **Closed - form solution**: Under certain conditions, gives the closed - form solution $ \theta^*=(C_{XX}^\dagger C_{XY})^* $ of the regression problem. ### Summary This paper successfully solves the problem of learning a linear operator in infinite - dimensional spaces by reformulating the infinite - dimensional linear regression problem as an inverse problem and using regularization techniques. The paper not only provides theoretical contributions but also provides powerful tools and methods for practical applications.

Learning linear operators: Infinite-dimensional regression as a well-behaved non-compact inverse problem

Statistical inverse learning problems with random observations

Statistical Learning Theory for Neural Operators

Minimax Optimal Kernel Operator Learning via Multilevel Training

A sparse optimization approach to infinite infimal convolution regularization

Least Squares Approximations in Linear Statistical Inverse Learning Problems

Gradient-Based Non-Linear Inverse Learning

Operator learning based on sparse high-dimensional approximation

A universal reproducing kernel Hilbert space for learning nonlinear systems operators

Spectral Function Space Learning and Numerical Linear Algebra Networks for Solving Linear Inverse Problems

Linear methods for non-linear inverse problems

Non-Euclidean Monotone Operator Theory and Applications

Learning the optimal Tikhonov regularizer for inverse problems

Learning sparsity-promoting regularizers for linear inverse problems

Learning truly monotone operators with applications to nonlinear inverse problems

Optimal low-rank approximations of posteriors for linear Gaussian inverse problems on Hilbert spaces

Inverse Problems with Learned Forward Operators

Learned Regularization for Inverse Problems: Insights from a Spectral Model

Learning nonlocal regularization operators

Learning Lipschitz Operators with respect to Gaussian Measures with Near-Optimal Sample Complexity

Generalized Convergence Rates Results for Linear Inverse Problems in Hilbert Spaces