Abstract:High dimensional data reduction techniques are provided by using partial least squares within deep learning. Our framework provides a nonlinear extension of PLS together with a disciplined approach to feature selection and architecture design in deep learning. This leads to a statistical interpretation of deep learning that is tailor made for predictive problems. We can use the tools of PLS, such as scree-plot, bi-plot to provide model diagnostics. Posterior predictive uncertainty is available using MCMC methods at the last layer. Thus we achieve the best of both worlds: scalability and fast predictive rule construction together with uncertainty quantification. Our key construct is to employ deep learning within PLS by predicting the output scores as a deep learner of the input scores. As with PLS our X-scores are constructed using SVD and applied to both regression and classification problems and are fast and scalable. Following Frank and Friedman 1993, we provide a Bayesian shrinkage interpretation of our nonlinear predictor. We introduce a variety of new partial least squares models: PLS-ReLU, PLS-Autoencoder, PLS-Trees and PLS-GP. To illustrate our methodology, we use simulated examples and the analysis of preferences of orange juice and predicting wine quality as a function of input characteristics. We also illustrate Brillinger's estimation procedure to provide the feature selection and data dimension reduction. Finally, we conclude with directions for future research.

Partial least squares for dependent data

Kernel partial least squares for stationary data

A Two-Step Estimation Method for Grouped Data with Connections to the Extended Growth Curve Model and Partial Least Squares Regression.

A Two-Step PLS Inspired Method for Linear Prediction with Group Effect

M-Dependence Approximation for Dependent Random Variables

[HDDA] sparse subspace constrained partial least squares

Deep Learning Partial Least Squares

Functional Partial Least-Squares: Optimal Rates and Adaptation

SIMPLS: An alternative approach to partial least squares regression

Sparse partial least squares regression for simultaneous dimension reduction and variable selection

Sparse-penalized deep neural networks estimator under weak dependence

The Degrees of Freedom of Partial Least Squares Regression

Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

Sparse higher order partial least squares for simultaneous variable selection, dimension reduction, and tensor denoising

Learning Local Dependence In Ordered Data

Application of copulas to improve covariance estimation for partial least squares

Penalized Partial Least Squares Based on B-Splines Transformations

Asymptotics of Linear Regression with Linearly Dependent Data

Integrative sparse partial least squares

Robust Partial Least Squares Using Low Rank and Sparse Decomposition

Least Squares Approximations in Linear Statistical Inverse Learning Problems