Abstract:Nowadays, data analysis in the world of Big Data is connected typically to data mining, descriptive or exploratory statistics, e.~g.\ cluster analysis, classification or regression analysis. Aside these techniques there is a huge area of methods from inferential statistics that are rarely considered in connection with Big Data. Nevertheless, inferential methods are also of use for Big Data analysis, especially for quantifying uncertainty. The article at hand will provide some insights to methodological and technical issues referring inferential methods in the Big Data area in order to bring together Big Data and inferential statistics, as it comes along with its difficulties. We present an approach that allows testing goodness-of-fit without model assumptions and relying on the empirical distribution. Especially, the method is able to utilize information from large datasets. Thereby, the approach is based on a clear theoretical background. We concentrate on the widely-used Kolmogorov-Smirnov test that is applied for testing goodness-of-fit in statistics. Our approach can be parallelized easily, which makes it applicable to distributed datasets particularly on a compute cluster. By this contribution, we turn to an audience that is interested in the technical and methodological backgrounds while implementing especially inferential statistical methods with Big Data tools as e. g. Spark.

Comparison of different goodness-of-fit tests

A new class of binning free, multivariate goodness-of-fit tests: the energy tests

Non-parametric Tests for the Tail Equivalence Via Empirical Likelihood

On goodness-of-fit tests for arbitrary multivariate models

A new set of tools for goodness-of-fit validation

A Viable Method for Goodness-of-fit Test in Maximum Likelihood Fit

Comparative evaluation of goodness of fit tests for normal distribution using simulation and empirical data

Asymptotically Distribution-Free Goodness-of-Fit Testing: A Unifying View

A new test for the multivariate two-sample problem based on the concept of minimum energy

Evaluating the Three Methods of Goodness of Fit Test for Frequency Analysis

A Goodness-of-Fit Test for Statistical Models

A power comparison of various normality tests

Tests of goodness of fit to multiple data sets

An overview of the goodness-of-fit test problem for copulas

Almost goodness-of-fit tests

Goodness-of-fit testing in bivariate count time series based on a bivariate dispersion index

Asymptotic Intermediate Efficiency of the Chi-square and Likelihood Ratio Goodness of Fit Tests

Gini index based goodness-of-fit test for the Lindley distribution

Two New Tests for Equality of Several Covariance Functions

Goodness-of-Fit Tests for Large Datasets

An Online Test for Goodness-of-Fit in Logistic Regression Model