Projected Gradient Descent Algorithms for Solving Nonlinear Inverse Problems with Generative Priors

Zhaoqiang Liu,Jun Han

DOI: https://doi.org/10.48550/arXiv.2209.10093

2022-09-21

Abstract:In this paper, we propose projected gradient descent (PGD) algorithms for signal estimation from noisy nonlinear measurements. We assume that the unknown $p$-dimensional signal lies near the range of an $L$-Lipschitz continuous generative model with bounded $k$-dimensional inputs. In particular, we consider two cases when the nonlinear link function is either unknown or known. For unknown nonlinearity, similarly to \cite{liu2020generalized}, we make the assumption of sub-Gaussian observations and propose a linear least-squares estimator. We show that when there is no representation error and the sensing vectors are Gaussian, roughly $O(k \log L)$ samples suffice to ensure that a PGD algorithm converges linearly to a point achieving the optimal statistical rate using arbitrary initialization. For known nonlinearity, we assume monotonicity as in \cite{yang2016sparse}, and make much weaker assumptions on the sensing vectors and allow for representation error. We propose a nonlinear least-squares estimator that is guaranteed to enjoy an optimal statistical rate. A corresponding PGD algorithm is provided and is shown to also converge linearly to the estimator using arbitrary initialization. In addition, we present experimental results on image datasets to demonstrate the performance of our PGD algorithms.

Machine Learning

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to solve the problem of estimating signals from noisy nonlinear measurements. Specifically, the paper considers two cases:

1. **Unknown nonlinear function**:
- Assume that the signal \( \mathbf{x}^* \) lies within the range of a generative model, and the generative model has \( L \)-Lipschitz continuity and bounded \( k \)-dimensional input.
- The unknown nonlinear function \( f \) may cause nonlinear distortion of the measured values.
- The paper proposes a projected gradient descent (PGD) algorithm to minimize the linear least - squares loss function and proves that in the case of Gaussian measurement vectors, approximately \( O(k \log L) \) samples are sufficient to ensure that the algorithm converges linearly to the optimal statistical rate.

2. **Known nonlinear function**:
- Assume that the nonlinear function \( f \) is known and monotonic.
- Representation errors in the signal are allowed, that is, the signal does not necessarily lie entirely within the range of the generative model.
- The paper proposes a nonlinear least - squares estimator and provides a corresponding PGD algorithm, proving that this algorithm can also converge linearly to the optimal statistical rate.

### Main contributions

1. **Case of unknown nonlinear function**:
- A PGD algorithm is proposed to minimize the linear least - squares loss function.
- It is proved that in the case of Gaussian measurement vectors, approximately \( O(k \log L) \) samples are sufficient to ensure that the algorithm converges linearly to the optimal statistical rate, which is approximately \( \sqrt{\frac{k \log L}{n}} \).
- Theoretical guarantees for choosing the step size are provided.

2. **Case of known nonlinear function**:
- A nonlinear least - squares estimator is proposed and the corresponding PGD algorithm is provided.
- It is proved that this algorithm can converge linearly to the optimal statistical rate and allows for representation errors in the signal.
- The proposed PGD algorithm is novel.

3. **Experimental results**:
- Various numerical experiments were carried out on image datasets to verify the validity of the theoretical results.

### Related work

- **Nonlinear inverse problems**: Previous research has mainly focused on low - dimensional settings, assuming that the measurement vector or link function satisfies certain conditions. For example, the maximum rank correlation estimator for monotonic link functions.
- **Inverse problems with generative priors**: In recent years, researchers have begun to use generative models instead of sparsity assumptions to reduce the number of required measurements.

### Mathematical notations

- Capital and lowercase bold letters are used to represent matrices and vectors respectively.
- \( \mathbf{G}: \mathbb{R}^k \to \mathbb{R}^p \) represents a generative model, where \( k \ll p \).
- \( \|\mathbf{X}\|_{2 \to 2} \) represents the spectral norm of the matrix \( \mathbf{X} \).
- \( \mathcal{B}_q^k(r) \) represents the \( \ell_q \) ball.

### Preliminaries

- **Sub - Gaussian random variables**: Defines the conditions for a random variable \( X \) to be sub - Gaussian and its sub - Gaussian norm.
- **Two - sided set restricted eigenvalue condition (TS - REC)**: Defines the conditions for a matrix \( \tilde{\mathbf{A}} \) to satisfy TS - REC.

### PGD algorithm for unknown nonlinear functions

- **Assumptions**:
- The signal \( \mathbf{x}^* \) lies within the range of the generative model.
- The measurement vectors \( \mathbf{a}_i \) are independent and identically distributed Gaussian vectors.
- The uncontaminated measurement values \( y_i \) satisfy the single - index model (SIM).
- The measurement values \( y_i \) are sub - Gaussian.
- Adversarial noise is allowed.
- **Optimization problem**: Minimize the linear \( \ell_2 \) loss function: