Abstract:We study the problem of missing data imputation, which is a fundamental task in the area of data quality that aims to impute the missing data to achieve the completeness of datasets. Though the recent distribution-modeling-based techniques (e.g., distribution generation and distribution matching) can achieve state-of-the-art performance in terms of imputation accuracy, we notice that (1) they deploy a sophisticated deep learning model that tends to be overfitting for missing data imputation; (2) they directly rely on a global data distribution while overlooking the local information. Driven by the inherent variability in both missing data and missing mechanisms, in this paper, we explore the uncertain nature of this task and aim to address the limitations of existing works by proposing an uNcertainty-driven netwOrk for Missing data Imputation, termed NOMI. NOMI has three key components, i.e., the retrieval module, the neural network gaussian process imputator (NNGPI) and the uncertainty-based calibration module. NOMI~ runs these components sequentially and in an iterative manner to achieve a better imputation performance. Specifically, in the retrieval module, NOMI~ retrieves local neighbors of the incomplete data samples based on the pre-defined similarity metric. Subsequently, we design NNGPI~ that merges the advantages of both the Gaussian Process and the universal approximation capacity of neural networks. NNGPI~ models the uncertainty by learning the posterior distribution over the data to impute missing values while alleviating the overfitting issue. Moreover, we further propose an uncertainty-based calibration module that utilizes the uncertainty of the imputator on its prediction to help the retrieval module obtain more reliable local information, thereby further enhancing the imputation performance. We also demonstrate that our NOMI~ can be reformulated as an instance of the well-known Expectation Maximization (EM) algorithm, highlighting the strong theoretical foundation of our proposed methods. Extensive experiments are conducted over 12 real-world datasets. The results demonstrate the excellent performance of NOMI in terms of both accuracy and efficiency.

Confidence Intervals Based on Survey Data with Nearest Neighbor Imputation

Compromised imputation based mean estimators using robust quantile regression

Uncertainty Quantification and Confidence Intervals for Naive Rare-Event Estimators

Optimal Nonparametric Inference with Two-Scale Distributional Nearest Neighbors

NIIA: Nonparametric Iterative Imputation Algorithm

Small Area Quantile Estimation

Exact adaptive confidence intervals for small areas

Two-Sample Inference in Highly Dispersed Negative Binomial Models

Numerical Data Imputation for Multimodal Data Sets: A Probabilistic Nearest-Neighbor Kernel Density Approach

Missing Data Imputation with Uncertainty-Driven Network

A Kernel-Based Conditional Two-Sample Test Using Nearest Neighbors (with Applications to Calibration, Regression Curves, and Simulation-Based Inference)

Confidence Intervals for Nonparametric Empirical Bayes Analysis

A novel ranked k-nearest neighbors algorithm for missing data imputation

Missing Data Imputation by Utilizing Information Within Incomplete Instances

Sample Correlation Coefficients Based on Survey Data under Regression Imputation

An Empirical Study of Dynamic Incomplete-Case Nearest Neighbor Imputation in Software Quality Data.

Cross-validation based K nearest neighbor imputation for software quality datasets: An empirical study.

A Fast Imputation Algorithm in Quantile Regression

Improved confidence intervals for nonlinear mixed-effects and nonparametric regression models

A Normality Test for High-dimensional Data Based on the Nearest Neighbor Approach

Network Imputation for a Spatial Autoregression Model with Incomplete Data