Abstract:Nowadays, data analysis in the world of Big Data is connected typically to data mining, descriptive or exploratory statistics, e.~g.\ cluster analysis, classification or regression analysis. Aside these techniques there is a huge area of methods from inferential statistics that are rarely considered in connection with Big Data. Nevertheless, inferential methods are also of use for Big Data analysis, especially for quantifying uncertainty. The article at hand will provide some insights to methodological and technical issues referring inferential methods in the Big Data area in order to bring together Big Data and inferential statistics, as it comes along with its difficulties. We present an approach that allows testing goodness-of-fit without model assumptions and relying on the empirical distribution. Especially, the method is able to utilize information from large datasets. Thereby, the approach is based on a clear theoretical background. We concentrate on the widely-used Kolmogorov-Smirnov test that is applied for testing goodness-of-fit in statistics. Our approach can be parallelized easily, which makes it applicable to distributed datasets particularly on a compute cluster. By this contribution, we turn to an audience that is interested in the technical and methodological backgrounds while implementing especially inferential statistical methods with Big Data tools as e. g. Spark.

On goodness-of-fit tests for arbitrary multivariate models

Goodness-of-fit testing in high-dimensional generalized linear models

Characteristic Function-Based Testing for Multifactor Continuous-Time Markov Models Via Nonparametric Regression

A Goodness-of-fit Test for Parametric and Semi-Parametric Models in Multiresponse Regression

Goodness-of-fit tests for general linear models with covariates missed at random

Goodness-Of-Fit Testing for Varying-Coefficient Models

A Goodness-of-Fit Test for Statistical Models

A general Monte Carlo method for multivariate goodness-of-fit testing applied to elliptical families

Comparison of different goodness-of-fit tests

Generalized spectral testing for multivariate continuous-time models

Data-driven goodness-of-fit tests

Testing Elliptical Models in High Dimensions

A new class of binning free, multivariate goodness-of-fit tests: the energy tests

Goodness-of-fit Testing in Linear Regression Models

Goodness-of-Fit Tests for Large Datasets

Integral Transform Methods in Goodness-of-Fit Testing, II: The Wishart Distributions

Almost goodness-of-fit tests

Asymptotically Distribution-Free Goodness-of-Fit Testing: A Unifying View

Tests of goodness of fit to multiple data sets

A new set of tools for goodness-of-fit validation

Goodness-of-Fit Assessment in Multidimensional Scaling and Unfolding