Abstract:We present a novel empirical approach toward measuring the Probability Density Function (PDF) of the deep features of Convolutional Neural Networks (CNNs). Measurement of the deep feature PDF is a valuable problem for several reasons. Notably, a. Understanding the deep feature PDF yields new insight into deep representations. b. Feature density methods are important for tasks such as anomaly detection which can improve the robustness of deep learning models in the wild. Interpretable measurement of the deep feature PDF is challenging due to the Curse of Dimensionality (CoD), and the Spatial intuition Limitation. Our novel measurement technique combines copula analysis with the Method of Orthogonal Moments (MOM), in order to directly measure the Generalized Characteristic Function (GCF) of the multivariate deep feature PDF. We find that, surprisingly, the one-dimensional marginals of non-negative deep CNN features after major blocks are not well approximated by a Gaussian distribution, and that these features increasingly approximate an exponential distribution with increasing network depth. Furthermore, we observe that deep features become increasingly independent with increasing network depth within their typical ranges. However, we surprisingly also observe that many deep features exhibit strong dependence (either correlation or anti-correlation) with other extremely strong detections, even if these features are independent within typical ranges. We elaborate on these findings in our discussion, where we propose a new hypothesis that exponentially infrequent large valued features correspond to strong computer vision detections of semantic targets, which would imply that these large-valued features are not outliers but rather an important detection signal.

Fourier Basis Density Model

A new approach to nonparametric estimation of multivariate spectral density function using basis expansion

Learning with Density Matrices and Random Features

Fourier PINNs: From Strong Boundary Conditions to Adaptive Fourier Bases

Fourier Neural Network Approximation of Transition Densities in Finance

Generalized multi-view model: Adaptive density estimation under low-rank constraints

Data-driven deep density estimation

Kernel Density Matrices for Probabilistic Deep Learning

Squared Neural Families: A New Class of Tractable Density Models

Interpretable Measurement of CNN Deep Feature Density using Copula and the Generalized Characteristic Function

Fourier Head: Helping Large Language Models Learn Complex Probability Distributions

Sparse Density Estimation with Measurement Errors

Density functionals from deep learning

Density Estimation via Binless Multidimensional Integration

Bayesian learning of joint distributions of objects

Principal component analysis in Bayes spaces for sparsely sampled density functions

Efficient spline orthogonal basis for representation of density functions

Learned Compression of Encoding Distributions

A flexible class of priors for orthonormal matrices with basis function-specific structure

A mixture of ellipsoidal densities for 3D data modelling

A Bayesian approach for parameter estimation and prediction using a computationally intensive model