Abstract:We study universal traits which emerge both in real-world complex datasets, as well as in artificially generated ones. Our approach is to analogize data to a physical system and employ tools from statistical physics and Random Matrix Theory (RMT) to reveal their underlying structure. We focus on the feature-feature covariance matrix, analyzing both its local and global eigenvalue statistics. Our main observations are: (i) The power-law scalings that the bulk of its eigenvalues exhibit are vastly different for uncorrelated normally distributed data compared to real-world data, (ii) this scaling behavior can be completely modeled by generating Gaussian data with long range correlations, (iii) both generated and real-world datasets lie in the same universality class from the RMT perspective, as chaotic rather than integrable systems, (iv) the expected RMT statistical behavior already manifests for empirical covariance matrices at dataset sizes significantly smaller than those conventionally used for real-world training, and can be related to the number of samples required to approximate the population power-law scaling behavior, (v) the Shannon entropy is correlated with local RMT structure and eigenvalues scaling, is substantially smaller in strongly correlated datasets compared to uncorrelated ones, and requires fewer samples to reach the distribution entropy. These findings show that with sufficient sample size, the Gram matrix of natural image datasets can be well approximated by a Wishart random matrix with a simple covariance structure, opening the door to rigorous studies of neural network dynamics and generalization which rely on the data Gram matrix.

Statistical Laws in Complex Systems

Data-Driven Automated Discovery of Variational Laws Hidden in Physical Systems

Statistical Patterns in the Equations of Physics and the Emergence of a Meta-Law of Nature

Power-Law Distributions in Empirical Data

Zipf's and Taylor's Laws

Zipf'S Law Leads to Heaps' Law: Analyzing Their Relation in Finite-Size Systems

A Mechanism Producing Power Law Etc. Distributions

POWER-LAW DISTRIBUTIONS BASED ON EXPONENTIAL DISTRIBUTIONS: LATENT SCALING, SPURIOUS ZIPF'S LAW, AND FRACTAL RABBITS

Dynamical approach to Zipf's law

CompLex: legal systems through the lens of complexity science

Emergence of power-law distributions in self-segregation reaction-diffusion processes

Discovering power laws in computer programs

Statistical physics of complex information dynamics

The statistical physics of Cities

Power-law distributions in binned empirical data

Big Data and Large Numbers. Interpreting Zipf's Law

Zipf's law, 1/f noise, and fractal hierarchy

The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets

Power laws in physics

Can statistical physics contribute to the science of economics?

Scaling and universality in animate and inanimate systems