Abstract:In high-dimensional regression problems, often a relatively small subset of the features are relevant for predicting the outcome, and methods that impose sparsity on the solution are popular. When multiple correlated outcomes are available (multitask), reduced rank regression is an effective way to borrow strength and capture latent structures that underlie the data. Our proposal is motivated by the UK Biobank population-based cohort study, where we are faced with large-scale, ultrahigh-dimensional features, and have access to a large number of outcomes (phenotypes)-lifestyle measures, biomarkers, and disease outcomes. We are hence led to fit sparse reduced-rank regression models, using computational strategies that allow us to scale to problems of this size. We use a scheme that alternates between solving the sparse regression problem and solving the reduced rank decomposition. For the sparse regression component we propose a scalable iterative algorithm based on adaptive screening that leverages the sparsity assumption and enables us to focus on solving much smaller subproblems. The full solution is reconstructed and tested via an optimality condition to make sure it is a valid solution for the original problem. We further extend the method to cope with practical issues, such as the inclusion of confounding variables and imputation of missing values among the phenotypes. Experiments on both synthetic data and the UK Biobank data demonstrate the effectiveness of the method and the algorithm. We present multiSnpnet package, available at http://github.com/junyangq/multiSnpnet that works on top of PLINK2 files, which we anticipate to be a valuable tool for generating polygenic risk scores from human genetic studies.

Scalable Econometrics on Big Data -- The Logistic Regression on Spark

Distributed Logistic Regression for Massive Data with Rare Events

Big Data analytics. Three use cases with R, Python and Spark

Goodness-of-Fit Tests for Large Datasets

A Survey on Spark Ecosystem for Big Data Processing

High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates

Scalable Semiparametric Spatio-temporal Regression for Large Data Analysis

SparkRDF: Elastic Discreted RDF Graph Processing Engine with Distributed Memory

Statistical Methods and Computing for Big Data

Usages of Spark Framework with Different Machine Learning Algorithms

Mastering Rare Event Analysis: Optimal Subsample Size in Logistic and Cox Regressions

Robust and efficient subsampling algorithms for massive data logistic regression

Scalable Estimation for Structured Additive Distributional Regression

Communication‐efficient distributed large‐scale sparse multinomial logistic regression

Real-time semiparametric regression for distributed data sets

Least-Square Approximation for a Distributed System.

Random Forests for Big Data

Least Squares Approximation for a Distributed System

Using large samples in econometrics

LARGE-SCALE MULTIVARIATE SPARSE REGRESSION WITH APPLICATIONS TO UK BIOBANK

Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions