Abstract:This dissertation develops novel inferential methods and theory for assessing uncertainty of modern statistical procedures unique to big data analysis. In particular, we mainly focus on four challenging aspects of big data: massive sample size, high dimensionality, heterogeneity and complexity. To begin with, we consider a partially linear framework for modeling massive heterogeneous data. The major goal is to extract common features across all sub-populations while exploring heterogeneity of each sub-population. In particular, we propose an aggregation type estimator for the commonality parameter that possesses the (non-asymptotic) minimax optimal bound and asymptotic distribution as if there were no heterogeneity. This oracle result holds when the number of sub-populations does not grow too fast. The next problem focuses on the challenge of the high dimensionality. We propose a robust inferential procedure for assessing uncertainties of parameter estimation in high dimensional linear models, where the dimension p can grow exponentially fast with the sample size n. We develop a new de-biasing framework tailored for nonsmooth loss functions. Our framework enables us to exploit the composite quantile function to construct a de-biased CQR estimator. This estimator is robust, and preserves efficiency in the sense that the worst case efficiency loss is less than 30% compared to square-loss-based procedures. In many cases our estimator is close to or better than the latter. Next, we consider the problem of high dimensional semiparametric generalized linear models. We propose a new inferential framework which addresses a variety of challenging problems in high dimensional data analysis, including incomplete data, selection bias, and heterogeneity. First, we develop a regularized statistical chromatography approach to infer the parameter of interest under the proposed semiparametric generalized linear model without the need of estimating the unknown base measure function. Then we propose a new likelihood ratio based framework iii to construct post-regularization confidence regions and tests for the low dimensional components of high dimensional parameters. We demonstrate the consequences of the general theory by using examples of missing data and multiple datasets inference. Lastly, we study the rank likelihood as a powerful inferential tool in multivariate analysis. The computation of the full rank likelihood function is often intractable in large-scale datasets. Motivated by this, we resort to lower order rank approximations and propose a new family of local rank likelihood functions. In particular, we show that the maximizer of the second-order local rank likelihood coincides with the Kendall’s tau correlation matrix for the transelliptical distribution family. Motivated by this new interpretation of the Kendall’s tau, we then investigate the third-order local rank likelihood, whose maximizer defines a new estimator that can be viewed as the third-order counterpart of the Kendall’s tau correlation matrix. We establish asymptotic normality and calculate its limiting variance under the Gaussian copula model, which enables the construction of confidence intervals based on this new estimator.

Statistical Inference and Large-scale Multiple Testing for High-dimensional Regression Models

Tests for a Multiple-Sample Problem in High Dimensions

Model-Free Statistical Inference on High-Dimensional Data

SIHR: Statistical Inference in High-Dimensional Linear and Logistic Regression Models

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

Shrinkage-based regularization tests for high-dimensional data with application to gene set analysis

SIHR: An R Package for Statistical Inference in High-dimensional Linear and Logistic Regression Models

Joint testing and false discovery rate control in high-dimensional multivariate regression.

Sparsified simultaneous confidence intervals for high-dimensional linear models

Confidence Intervals and Hypothesis Testing for High-Dimensional Regression

Testing Regression Coefficients in High-Dimensional and Sparse Settings

Two-Sample Tests for High-Dimensional Linear Regression with an Application to Detecting Interactions.

Confidence Intervals for Low-Dimensional Parameters in High-Dimensional Linear Models

Recent Advances in Statistical Methodologies in Evaluating Program for High-Dimensional Data

Statistical Inference for Big Data

Integrative High Dimensional Multiple Testing with Heterogeneity under Data Sharing Constraints

Testing covariates in high-dimensional regression

A New Nonparametric Test for High-Dimensional Regression Coefficients

Testing a Single Regression Coefficient in High Dimensional Linear Models

High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

Statistical Inference for Genetic Relatedness Based on High-Dimensional Logistic Regression