Abstract:Tree tensor networks, or tree-based tensor formats, are prominent model classes for the approximation of high-dimensional functions in computational and data science. They correspond to sum-product neural networks with a sparse connectivity associated with a dimension tree and widths given by a tuple of tensor ranks. The approximation power of these models has been proved to be (near to) optimal for classical smoothness classes. However, in an empirical risk minimization framework with a limited number of observations, the dimension tree and ranks should be selected carefully to balance estimation and approximation errors. We propose and analyze a complexity-based model selection method for tree tensor networks in an empirical risk minimization framework and we analyze its performance over a wide range of smoothness classes. Given a family of model classes associated with different trees, ranks, tensor product feature spaces and sparsity patterns for sparse tensor networks, a model is selected (à la Barron, Birgé, Massart) by minimizing a penalized empirical risk, with a penalty depending on the complexity of the model class and derived from estimates of the metric entropy of tree tensor networks. This choice of penalty yields a risk bound for the selected predictor. In a least-squares setting, after deriving fast rates of convergence of the risk, we show that our strategy is (near to) minimax adaptive to a wide range of smoothness classes including Sobolev or Besov spaces (with isotropic, anisotropic or mixed dominating smoothness) and analytic functions. We discuss the role of sparsity of the tensor network for obtaining optimal performance in several regimes. In practice, the amplitude of the penalty is calibrated with a slope heuristics method. Numerical experiments in a least-squares regression setting illustrate the performance of the strategy.

Some upper bounds for the rate of convergence of penalized likelihood context tree estimators

Exponential inequalities for empirical unbounded context trees

Rate Of Convergence For The Maximum Likelihood Estimator In Fractional Ornstein-Uhlenbeck Processes

Phase transition on the convergence rate of parameter estimation under an Ornstein-Uhlenbeck diffusion on a tree

A UNIFIED STUDY OF NONPARAMETRIC INFERENCE FOR MONOTONE FUNCTIONS

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

Lower bounds on the rate of convergence for accept-reject-based Markov chains in Wasserstein and total variation distances

Approximate group context tree

Quasi-maximum likelihood estimation and penalized estimation under non-standard conditions

Context-tree weighting and Bayesian Context Trees: Asymptotic and non-asymptotic justifications

Bounds on the Speed and on Regeneration Times for Certain Processes on Regular Trees

Binomial-tree approximation for time-inconsistent stopping

Asymptotic Properties of Penalized Spline Estimators in Concave Extended Linear Models: Rates of Convergence

Generalizing Tree Probability Estimation Via Bayesian Networks

A Note on the Convergence Rates for Empirical Bayes Estimators of Parameters in Multiple-Parameter Exponential Families

Wasserstein-based Minimax Estimation of Dependence in Multivariate Regularly Varying Extremes

Berry-Esseen estimates for regenerative processes under weak moment assumptions

Learning with tree tensor networks: complexity estimates and model selection

Wasserstein-p Bounds in the Central Limit Theorem Under Local Dependence

Improvement of convergence rates for the empirical Bayes test problem in scale exponential family

Better-than-KL PAC-Bayes Bounds