Abstract:Neural networks typically exhibit permutation symmetries which contribute to the non-convexity of the networks' loss landscapes, since linearly interpolating between two permuted versions of a trained network tends to encounter a high loss barrier. Recent work has argued that permutation symmetries are the only sources of non-convexity, meaning there are essentially no such barriers between trained networks if they are permuted appropriately. In this work, we refine these arguments into three distinct claims of increasing strength. We show that existing evidence only supports "weak linear connectivity"-that for each pair of networks belonging to a set of SGD solutions, there exist (multiple) permutations that linearly connect it with the other networks. In contrast, the claim "strong linear connectivity"-that for each network, there exists one permutation that simultaneously connects it with the other networks-is both intuitively and practically more desirable. This stronger claim would imply that the loss landscape is convex after accounting for permutation, and enable linear interpolation between three or more independently trained models without increased loss. In this work, we introduce an intermediate claim-that for certain sequences of networks, there exists one permutation that simultaneously aligns matching pairs of networks from these sequences. Specifically, we discover that a single permutation aligns sequences of iteratively trained as well as iteratively pruned networks, meaning that two networks exhibit low loss barriers at each step of their optimization and sparsification trajectories respectively. Finally, we provide the first evidence that strong linear connectivity may be possible under certain conditions, by showing that barriers decrease with increasing network width when interpolating among three networks.

A variety of globally stable periodic orbits in permutation binary neural networks

Permutation binary neural networks: Analysis of periodic orbits and its applications

Analysis and FPGA based Implementation of Permutation Binary Neural Networks

Multiperiodicity of Periodically Oscillated Discrete-Time Neural Networks with Transient Excitatory Self-Connections and Sigmoidal Nonlinearities

Convergence Analysis of General Neural Networks under Almost Periodic Stimuli

Self-excitation of Neurons Leads to Multiperiodicity of Discrete-Time Neural Networks with Distributed Delays

The Existence and Global Attractivity of Almost Periodic Sequence Solution of Discrete-Time Neural Networks

Analysis of Cyclic Dynamics for Networks of Linear Threshold Neurons.

Simultaneous linear connectivity of neural networks modulo permutation

Stability and Bifurcation Analysis in a Class of Two-Neuron Networks with Resonant Bilinear Terms

Graph Theoretical Characterizations of Bit Permutation Networks

Classification of Periodic Variable Stars with Novel Cyclic-Permutation Invariant Neural Networks

Existence and global exponential stability of anti-periodic solutions for competitive neural networks with delays in the leakage terms on time scales

Permutation-equivariant neural networks applied to dynamics prediction

On permutation symmetries in Bayesian neural network posteriors: a variational perspective

Existence and Global Stability Analysis of Almost Periodic Solutions for Cohen-Grossberg Neural Networks

Global Exponential Stability of Neural Networks'Periodic Solution Involving Finite Distributed Delays with Periodic Inputs

Periodicity and global exponential periodic synchronization of delayed neural networks with discontinuous activations and impulsive perturbations

Multistability and new attraction basins of almost-periodic solutions of delayed neural networks.

Multiple Switching and Bifurcations of In-Phase and Anti-Phase Periodic Orbits to Chaotic Coexistence in a Delayed Half-Center CPG Oscillator

On permutation-invariant neural networks