Abstract:Observational data with massive sample sizes are often distributed on many local machines. From an experimental design perspective, investigators often desire to identify the effect of new treatments (even ML algorithms) on many blocks of experimental data. With time requirements or budget constraints, assigning all treatments to each block is not always feasible. This creates incomplete responses with respect to a randomized complete block design (RCBD). These incomplete responses are missing by design. However, whether they can be estimated with missing imputation methods is not well understood. Thus, it is challenging to correctly identify the treatment effects with missing data. To this end, this paper provides a method for imputation and analysis of the responses with missing data. The proposed method consists of three steps: Reconstruction, Imputation, and `Complete'-data Analysis (RICA). The incomplete responses are imputed with the expectation-maximization (EM) algorithm. The RCBD model is then fitted by the resulting dataset. The identifiability result suggests that the missing may be nonignorable for each block, but the whole data of an incomplete design are missing by design when the design is balanced. Theoretical results on relative efficiency also inform us when the missingness should be imputed for incomplete designs with the role of balanced variance. Applications on real-world data verify the efficacy of this method.

Characterizing the Estimation of Parameters in Incomplete-Data Problems

On Jointly Estimating Parameters and Missing Data by Maximizing the Complete-Data Likelihood

The Analysis of Social-Science Data with Missing Values

Statistical Data, Missing

Missing Data Imputation in Balanced Construction for Incomplete Block Designs

Conditions for Ignoring the Missing-Data Mechanism in Likelihood Inferences for Parameter Subsets

Missing Data Imputation: Focusing on Single Imputation.

Imputing Missing Data by Fully Conditional Models : Some Cautionary Examples and Guidelines

19 Incomplete Data in Epidemiology and Medical Statistics

Statistical Analysis with Missing Data

In-Database Data Imputation

Inference for Large Dimensional Factor Models under General Missing Data Patterns

Nonparametric Statistical Inference and Imputation for Incomplete Categorical Data

Automated Learning of Factor Analysis with Complete and Incomplete Data.

Estimation in Semiparametric Models with Missing Data

Empirical Likelihood for Estimating Equations with Missing Values

Factorization of posteriors and partial imputation algorithm for graphical models with missing data

Nonparametric Imputation of Missing Values for Estimating Equation Based Inference

Robust Bayesian Classification with Incomplete Data

Mining Incomplete Data Using Global and Saturated Probabilistic Approximations Based on Characteristic Sets and Maximal Consistent Blocks

An Imputation-Consistency Algorithm for High-Dimensional Missing Data Problems and Beyond