Abstract:We develop a general theory of synaptic neural balance and how it can emerge or be enforced in neural networks. For a given regularizer, a neuron is said to be in balance if the total cost of its input weights is equal to the total cost of its output weights. The basic example is provided by feedforward networks of ReLU units trained with $L_2$ regularizers, which exhibit balance after proper training. The theory explains this phenomenon and extends it in several directions. The first direction is the extension to bilinear and other activation functions. The second direction is the extension to more general regularizers, including all $L_p$ regularizers. The third direction is the extension to non-layered architectures, recurrent architectures, convolutional architectures, as well as architectures with mixed activation functions. Gradient descent on the error function alone does not converge in general to a balanced state, where every neuron is in balance, even when starting from a balanced state. However, gradient descent on the regularized error function ought to converge to a balanced state, and thus network balance can be used to assess learning progress. The theory is based on two local neuronal operations: scaling which is commutative, and balancing which is not commutative. Given any initial set of weights, when local balancing operations are applied to each neuron in a stochastic manner, global order always emerges through the convergence of the stochastic balancing algorithm to the same unique set of balanced weights. The reason for this is the existence of an underlying strictly convex optimization problem where the relevant variables are constrained to a linear, only architecture-dependent, manifold. Simulations show that balancing neurons prior to learning, or during learning in alternation with gradient descent steps, can improve learning speed and final performance.

Spatially extended balanced networks without translationally invariant connectivity

Balanced Networks of Spiking Neurons with Spatially Dependent Recurrent Connections

Memory Storage and Retrieval in Sparsely Connected Balanced Networks

Highly connected neurons spike less frequently in balanced networks

Spatially Extended Networks with Singular Multi-scale Connectivity Patterns

A biophysical model of dynamic balancing of excitation and inhibition in fast oscillatory large-scale networks

Synaptic Plasticity in Correlated Balanced Networks

Nonnormal amplification in random balanced neuronal networks

Emergence of a Balanced Core through Dynamical Competition in Heterogeneous Neuronal Networks

Compressive Sensing Inference of Neuronal Network Connectivity in Balanced Neuronal Dynamics

The Hydrodynamic Limit of Neural Networks with Balanced Excitation and Inhibition

Network algorithmics and the emergence of the cortical synaptic-weight distribution

Balanced Active Core in Heterogeneous Neuronal Networks

Emergence of Irregular Activity in Networks of Strongly Coupled Conductance-Based Neurons

Synchronization of the Small-World Neuronal Network with Unreliable Synapses.

The Impact of Spike-Frequency Adaptation on Balanced Network Dynamics

Effects of Spatial Constraints of Inhibitory Connectivity on the Dynamical Development of Criticality in Spiking Networks

A robust balancing mechanism for spiking neural networks

A Theory of Synaptic Neural Balance: From Local to Global Order

Extreme-value statistics of brain networks: importance of balanced condition

Robust spatial memory maps encoded in networks with transient connections