Abstract:This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparameterized two-layer neural networks. In particular, we consider the minimax optimization problem stemming from estimating linear functional equations defined by conditional expectations, where the objective functions are quadratic in the functional spaces. We address (i) the convergence of the stochastic gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. We establish convergence under the mean-field regime by considering the continuous-time and infinite-width limit of the optimization dynamics. Under this regime, the stochastic gradient descent-ascent corresponds to a Wasserstein gradient flow over the space of probability measures defined over the space of neural network parameters. We prove that the Wasserstein gradient flow converges globally to a stationary point of the minimax objective at a $O(T^{-1} + \alpha^{-1})$ sublinear rate, and additionally finds the solution to the functional equation when the regularizer of the minimax objective is strongly convex. Here $T$ denotes the time and $\alpha$ is a scaling parameter of the neural networks. In terms of representation learning, our results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(\alpha^{-1})$, measured in terms of the Wasserstein distance. Finally, we apply our general results to concrete examples including policy evaluation, nonparametric instrumental variable regression, asset pricing, and adversarial Riesz representer estimation.

Second-Order Characterization of Minimax Parameter Estimation in Restricted Parameter Space

On Asymptotic Properties of the Parameter Estimator for a Type of SPDE

Minimax estimation of linear functionals under squared error loss

Characterizing the minimax rate of nonparametric regression under bounded convex constraints

Two New Families of Local Asymptotically Minimax Lower Bounds in Parameter Estimation

Identifiability Implies Robust, Globally Exponentially Convergent On-line Parameter Estimation: Application to Model Reference Adaptive Control

Local Asymptotic Minimax Estimation of Nonregular Parameters with Translation-Scale Equivariant Maps

Mini-Max-Risk and Mini-Mean-Risk Inferences for A Partially Piecewise Regression

Noisy recovery from random linear observations: Sharp minimax rates under elliptical constraints

Signal-to-noise ratio aware minimaxity and higher-order asymptotics

Towards Sharper Risk Bounds for Minimax Problems

An elementary approach for minimax estimation of Bernoulli proportion in the restricted parameter space

High-probability minimax lower bounds

A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization

Wasserstein-based Minimax Estimation of Dependence in Multivariate Regularly Varying Extremes

Constructive and consistent estimation of quadratic minimax

The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy

Finding Second-Order Stationary Point for Nonconvex-Strongly-Concave Minimax Problem

Parameter estimation for second-order SPDEs in multiple space dimensions

A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimiax Optimization

Minimax Rates for High-dimensional Double Sparse Structure over $\ell_q$-balls