Abstract:This paper introduces a novel approach to learning sparsity-promoting regularizers for solving linear inverse problems. We develop a bilevel optimization framework to select an optimal synthesis operator, denoted as $B$, which regularizes the inverse problem while promoting sparsity in the solution. The method leverages statistical properties of the underlying data and incorporates prior knowledge through the choice of $B$. We establish the well-posedness of the optimization problem, provide theoretical guarantees for the learning process, and present sample complexity bounds. The approach is demonstrated through examples, including compact perturbations of a known operator and the problem of learning the mother wavelet, showcasing its flexibility in incorporating prior knowledge into the regularization framework. This work extends previous efforts in Tikhonov regularization by addressing non-differentiable norms and proposing a data-driven approach for sparse regularization in infinite dimensions.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to solve the sparse regularization problem in linear inverse problems. Specifically, the author introduced a new method to learn sparsity - promoting regularizers for solving linear inverse problems. The core idea is to select an optimal synthesis operator $B$ through a bilevel optimization framework, which can not only regularize the inverse problem but also promote sparsity in the solution. #### Specific problem description Consider a linear inverse problem: \[y = Ax+\varepsilon,\] where $A: X\rightarrow Y$ is a bounded linear operator between Hilbert spaces $X$ and $Y$, and the inverse operator $A^{-1}$ (if it exists) is usually an unbounded operator. To regularize this inverse problem and simultaneously promote the sparsity of the solution, the author proposed the following minimization problem: \[\hat{x}_B = B\hat{u}_B,\quad\hat{u}_B=\arg\min_{u\in\ell^2}\left\{\frac{1}{2}\|\Sigma_{\varepsilon}^{-1/2}ABu\|_Y^2-\langle y,\Sigma_{\varepsilon}^{-1}ABu\rangle_Y+\|u\|_{\ell^1}\right\},\] where $\Sigma_{\varepsilon}$ is the covariance matrix of the noise $\varepsilon$. ### Main contributions 1. **Research on theoretical properties**: Analyzed the well - posedness of the inner minimization problem (formula (2)), especially the continuous dependence of the minimum solution $\hat{x}_B$ for a fixed $B$. 2. **Approximation properties of empirical objectives**: Studied the approximation properties of the empirical objective $\hat{B}$ based on sample data to the optimal objective $B^{\star}$, and derived sample error estimates. 3. **Application examples**: Provided application examples that satisfy the assumptions, demonstrating the flexibility and effectiveness of this method in different scenarios. ### Innovation points - Proposed a new bilevel optimization framework for selecting the optimal synthesis operator $B$ to achieve sparse regularization. - Applied statistical learning methods to infinite - dimensional sparse optimization problems, extended the traditional Tikhonov regularization method, and solved the non - differentiable norm problem. - Provided strict theoretical guarantees and sample complexity bounds to ensure the reliability and effectiveness of the method. ### Key formulas - Linear inverse problem model: \[y = Ax+\varepsilon\] - Minimization problem: \[\hat{u}_B=\arg\min_{u\in\ell^2}\left\{\frac{1}{2}\|\Sigma_{\varepsilon}^{-1/2}ABu\|_Y^2-\langle y,\Sigma_{\varepsilon}^{-1}ABu\rangle_Y+\|u\|_{\ell^1}\right\}\] - Expected loss function: \[L(B)=\mathbb{E}_{(x,y)\sim\rho}\left[\|R_B(y) - x\|_X^2\right]\] - Empirical risk: \[\hat{L}(B)=\frac{1}{m}\sum_{j = 1}^m\|R_B(y_j)-x_j\|_X^2\] Through these formulas and methods, the paper provides a data - driven sparse regularization strategy, which is suitable for high - dimensional and infinite - dimensional linear inverse problems.

Learning sparsity-promoting regularizers for linear inverse problems

Learning Sparsity-Promoting Regularizers Using Bilevel Optimization

Learning the optimal Tikhonov regularizer for inverse problems

A sparse optimization approach to infinite infimal convolution regularization

Supervised Learning of Sparsity-Promoting Regularizers for Denoising

Learned Regularization for Inverse Problems: Insights from a Spectral Model

On Learning the Optimal Regularization Parameter in Inverse Problems

Learning nonlocal regularization operators

Iterative regularization for low complexity regularizers

Derivative-free stochastic bilevel optimization for inverse problems

Subspace projection regularization for large-scale Bayesian linear inverse problems

A multilevel framework for sparse optimization with application to inverse covariance estimation and logistic regression

Sparse Regularization Based on Orthogonal Tensor Dictionary Learning for Inverse Problems

Smoothing $\mathcal{L}^2$ gradients in iterative regularization

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

Learning a Gaussian Mixture for Sparsity Regularization in Inverse Problems

Automatic nonstationary anisotropic Tikhonov regularization through bilevel optimization

High-dimensional Inference Via Lipschitz Sparsity-Yielding Regularizers.

Shearlet-based regularization in statistical inverse learning with an application to X-ray tomography

Fictitious null spaces for improving the solution of injective inverse problems

Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithms