Abstract:We consider the classical \emph{spherical} perceptrons and study their capacities. The famous zero-threshold case was solved in the sixties of the last century (see, \cite{Wendel62,Winder,Cover65}) through the high-dimensional combinatorial considerations. The general threshold, $\kappa$, case though turned out to be much harder and stayed out of reach for the following several decades. A substantial progress was then made in \cite{SchTir02} and \cite{StojnicGardGen13} where the \emph{positive} threshold ($\kappa\geq 0$) scenario was finally fully settled. While the negative counterpart ($\kappa\leq 0$) remained out of reach, \cite{StojnicGardGen13} did show that the random duality theory (RDT) is still powerful enough to provide excellent upper bounds. Moreover, in \cite{StojnicGardSphNeg13}, a \emph{partially lifted} RDT variant was considered and it was shown that the upper bounds of \cite{StojnicGardGen13} can be lowered. After recent breakthroughs in studying bilinearly indexed (bli) random processes in \cite{Stojnicsflgscompyx23,Stojnicnflgscompyx23}, \emph{fully lifted} random duality theory (fl RDT) was developed in \cite{Stojnicflrdt23}. We here first show that the \emph{negative spherical perceptrons} can be fitted into the frame of the fl RDT and then employ the whole fl RDT machinery to characterize the capacity. To be fully practically operational, the fl RDT requires a substantial numerical work. We, however, uncover remarkable closed form analytical relations among key lifting parameters. Such a discovery enables performing the needed numerical calculations to obtain concrete capacity values. We also observe that an excellent convergence (with the relative improvement $\sim 0.1\%$) is achieved already on the third (second non-trivial) level of the \emph{stationarized} full lifting.

Fixed width treelike neural networks capacity analysis -- generic activations

Exact capacity of the \emph{wide} hidden layer treelike neural networks with generic activations

Capacity of the treelike sign perceptrons neural networks with one hidden layer -- RDT based upper bounds

\emph{Lifted} RDT based capacity analysis of the 1-hidden layer treelike \emph{sign} perceptrons neural networks

Memory capacity of two layer neural networks with smooth activations

Capacity bounds for hyperbolic neural network representations of latent tree structures

Solution space and storage capacity of fully connected two-layer neural networks with generic activation functions

Binary perceptrons capacity via fully lifted random duality theory

Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations

Fl RDT based ultimate lowering of the negative spherical perceptron capacity

Injectivity capacity of ReLU gates

Memory Capacity of Recurrent Neural Networks with Matrix Representation

A Capacity Scaling Law for Artificial Neural Networks

Single-unit activations confer inductive biases for emergent circuit solutions to cognitive tasks

A Tropical Approach to Neural Networks with Piecewise Linear Activations

Latent Assistance Networks: Rediscovering Hyperbolic Tangents in RL

Towards Lower Bounds on the Depth of ReLU Neural Networks

Exponential Capacity of Dense Associative Memories

Capacity of the Hebbian-Hopfield network associative memory

Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK

Implicit Hypersurface Approximation Capacity in Deep ReLU Networks