Abstract:The correlation coefficient is one of the essential statistical techniques used to discover relationships among variables. Various techniques can quantify correlation, such as Pearson's, Spearman's, and Kendall's correlation coefficients, depending on the data type. As with any use of data, missing data will impact the availability of data, reducing it and potentially affecting the results. Furthermore, the removal of missing-value data from the study when using complete case analysis or available case analysis may result in selection biases. In this paper, we investigate the impact of missing data on the correlation coefficient value by calculating the difference between the correlation coefficient of the original complete dataset and that of a dataset with missing data. Two deletion strategies (Listwise and Pairwise) and three imputation strategies (Mean, k-Nearest Neighbors (k-NN), and Expectation-Maximization) were used to prepare the data before calculating the correlation coefficient. Unique correlation coefficient values were created by converting unique values to a one-dimensional array, and RMSE metrics were used to evaluate the experiments. Eight UCI and Kaggle datasets with different sizes and numbers of attributes were used in this study. The experiment results demonstrate that the Pairwise strategy and k-NN give good results on the correlation coefficient, respectively, when the missing rate is moderate or less. Pairwise uses all the available values and discards only the missing values of the related attribute, while k-NN fills the missing values with new values that produce correlation coefficient values close to the actual values.

Comparing Regressions when Some Predictor Values Are Missing

SELECTING THE “BEST” REGRESSION WHEN FACED WITH MISSING OBSERVATIONS

Regression with Highly Correlated Predictors: Variable Omission Is Not the Solution

Impact of Missing Data on Correlation Coefficient Values: Deletion and Imputation Methods for Data Preparation

Pearson's correlation under the scope: Assessment of the efficiency of Pearson's correlation to select predictor variables for linear models

Missing values handling for machine learning portfolios

Sample Correlation Coefficients Based on Survey Data under Regression Imputation

Regression with missing Ys: An improved strategy for analyzing multiply imputed data

Comparing Correlated but Nonoverlapping Correlations.

The Analysis of Social-Science Data with Missing Values

Correlation visualization under missing values: a comparison between imputation and direct parameter estimation methods

Imputation for prediction: beware of diminishing returns

Auxiliary variables in multiple imputation in regression with missing X: a warning against including too many in small sample research

On the Relation between Prediction and Imputation Accuracy under Missing Covariates

Are alternative variables in a set differently associated with a target variable? Statistical tests and practical advice for dealing with dependent correlations

The More Data, the Better? Demystifying Deletion-Based Methods in Linear Regression with Missing Data

Multicollinearity and Regression Analysis

A Comparison of Three Popular Methods for Handling Missing Data: Complete-Case Analysis, Inverse Probability Weighting, and Multiple Imputation

A Correlated Random Effects Model for Nonignorable Missing Data in Value-Added Assessment of Teacher Effects

Predictive Regressions: Method of Least Autocorrelation

Predicting Multivariate Responses in Multiple Linear Regression