Arbitrage equilibrium and the emergence of universal microstructure in deep neural networks

Venkat Venkatasubramanian,N Sanjeevrajan,Manasi Khandekar,Abhishek Sivaram,Collin Szczepanski

2024-06-05

Abstract:Despite the stunning progress recently in large-scale deep neural network applications, our understanding of their microstructure, 'energy' functions, and optimal design remains incomplete. Here, we present a new game-theoretic framework, called statistical teleodynamics, that reveals important insights into these key properties. The optimally robust design of such networks inherently involves computational benefit-cost trade-offs that are not adequately captured by physics-inspired models. These trade-offs occur as neurons and connections compete to increase their effective utilities under resource constraints during training. In a fully trained network, this results in a state of arbitrage equilibrium, where all neurons in a given layer have the same effective utility, and all connections to a given layer have the same effective utility. The equilibrium is characterized by the emergence of two lognormal distributions of connection weights and neuronal output as the universal microstructure of large deep neural networks. We call such a network the Jaynes Machine. Our theoretical predictions are shown to be supported by empirical data from seven large-scale deep neural networks. We also show that the Hopfield network and the Boltzmann Machine are the same special case of the Jaynes Machine.

Disordered Systems and Neural Networks

What problem does this paper attempt to address?

This paper investigates the microstructure, energy function, and incomplete understanding of optimal design of deep neural networks (DNNs). The author proposes a new game-theoretic framework called statistical dynamic learning to reveal new insights into these key properties. During the training process, neurons and connections compete to improve their utility under resource constraints, ultimately reaching an arbitrage equilibrium state where all neurons and connections within the same layer have the same utility. This equilibrium state is characterized by log-normal distributions of connection weights and neuron output features, and it is referred to as the general microstructure of large-scale DNNs. The author refers to such networks as Jaynes Machines and demonstrates that this theoretical prediction matches empirical data from seven large DNNs. Additionally, they point out that Hopfield networks and Boltzmann machines are special cases of Jaynes Machines. In the paper, the author also analyzes the distribution of connection weights, predicts a log-normal distribution of weights in sufficiently trained networks, and discusses how this distribution relates to the robustness and efficiency of the network.