Abstract:High dimensional data reduction techniques are provided by using partial least squares within deep learning. Our framework provides a nonlinear extension of PLS together with a disciplined approach to feature selection and architecture design in deep learning. This leads to a statistical interpretation of deep learning that is tailor made for predictive problems. We can use the tools of PLS, such as scree-plot, bi-plot to provide model diagnostics. Posterior predictive uncertainty is available using MCMC methods at the last layer. Thus we achieve the best of both worlds: scalability and fast predictive rule construction together with uncertainty quantification. Our key construct is to employ deep learning within PLS by predicting the output scores as a deep learner of the input scores. As with PLS our X-scores are constructed using SVD and applied to both regression and classification problems and are fast and scalable. Following Frank and Friedman 1993, we provide a Bayesian shrinkage interpretation of our nonlinear predictor. We introduce a variety of new partial least squares models: PLS-ReLU, PLS-Autoencoder, PLS-Trees and PLS-GP. To illustrate our methodology, we use simulated examples and the analysis of preferences of orange juice and predicting wine quality as a function of input characteristics. We also illustrate Brillinger's estimation procedure to provide the feature selection and data dimension reduction. Finally, we conclude with directions for future research.

Some theoretical aspects of partial least squares regression

Maximum Likelihood Estimators in a Two Step Model for PLS

A Two-Step PLS Inspired Method for Linear Prediction with Group Effect

Boosting the Partial Least Square Algorithm for Regression Modelling

A non-asymptotic analysis of the single component PLS regression

Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

A test of significance for partial least squares regression

SIMPLS: An alternative approach to partial least squares regression

Some recent developments in PLS modeling

PLS-regression: a basic tool of chemometrics

Using the Krylov subspace formulation to improve regularisation and interpretation in partial least squares regression

Partial least trimmed squares regression

[HDDA] sparse subspace constrained partial least squares

Model selection for partial least squares regression

Partial Least Squares (PLS) Methods: Origins, Evolution, and Application to Social Sciences

Functional Partial Least-Squares: Optimal Rates and Adaptation

Deep Learning Partial Least Squares

Robust methods for partial least squares regression

Penalized Partial Least Squares Based on B-Splines Transformations

Probabilistic Learning of Partial Least Squares Regression Model: Theory and Industrial Applications

Consistent Partial Least Squares for Nonlinear Structural Equation Models