Symmetry Breaking and Equivariant Neural Networks

Sékou-Oumar Kaba,Siamak Ravanbakhsh

2023-12-14

Abstract:Using symmetry as an inductive bias in deep learning has been proven to be a principled approach for sample-efficient model design. However, the relationship between symmetry and the imperative for equivariance in neural networks is not always obvious. Here, we analyze a key limitation that arises in equivariant functions: their incapacity to break symmetry at the level of individual data samples. In response, we introduce a novel notion of 'relaxed equivariance' that circumvents this limitation. We further demonstrate how to incorporate this relaxation into equivariant multilayer perceptrons (E-MLPs), offering an alternative to the noise-injection method. The relevance of symmetry breaking is then discussed in various application domains: physics, graph representation learning, combinatorial optimization and equivariant decoding.

Machine Learning

What problem does this paper attempt to address?

This paper discusses the effectiveness of using symmetry as prior knowledge in deep learning to design models, but also points out a limitation: equivariant functions cannot break symmetry at the individual data sample level. The authors introduce a new concept called "relaxed equivariance" to overcome this limitation. They demonstrate how to integrate this relaxation into equivariant multilayer perceptrons (E-MLPs), allowing the model to handle symmetry breaking while preserving symmetry. Key points of the paper include: 1. The importance of symmetry in data distributions, which naturally guides learning functions with equivariance constraints. 2. The limitation of equivariant functions: they cannot break symmetry at the individual sample level, which is necessary in physical systems such as phase transitions. 3. Introducing the concept of "relaxed equivariance" to allow the model to break symmetry on inputs and map to arbitrary orbit types. 4. Demonstrating how to construct E-MLPs that can handle symmetry breaking, as an alternative to noise injection methods. 5. Discussing applications in multiple domains such as physics, graph representation learning, combinatorial optimization, and equivariant decoding. 6. Providing directions for future research, including experimental validation and the potential of applying relaxed equivariance in different domains. In summary, this paper proposes a new framework to handle symmetry breaking in deep learning by relaxing equivariance to enhance the model's expressive power.

Symmetry Breaking and Equivariant Neural Networks

Breaking the Symmetry: Resolving Symmetry Ambiguities in Equivariant Neural Networks

Equivariant Symmetry Breaking Sets

Symmetry From Scratch: Group Equivariance as a Supervised Learning Task

Using and Abusing Equivariance

Theory for Equivariant Quantum Neural Networks

Relaxed Equivariant Graph Neural Networks

On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory

On the hardness of learning under symmetries

Symmetry Breaking in Neural Network Optimization: Insights from Input Dimension Expansion

Symmetry Perception by Deep Networks: Inadequacy of Feed-Forward Architectures and Improvements with Recurrent Connections

Approximately Equivariant Neural Processes

Symmetry Induces Structure and Constraint of Learning

Learning Broken Symmetries with Resimulation and Encouraged Invariance

Symmetry Breaking in Neuroevolution: A Technical Report

Equivariant neural networks for inverse problems

Symmetry Discovery for Different Data Types

Equivariant neural networks and piecewise linear representation theory

A Unified Framework to Enforce, Discover, and Promote Symmetry in Machine Learning

Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks

What is an equivariant neural network?