Abstract: Geometric deep learning, i.e., designing neural networks to handle the ubiquitous geometric data such as point clouds and graphs, have achieved great successes in the last decade. One critical inductive bias is that the model can maintain invariance towards various transformations such as translation, rotation, and scaling. The existing graph neural network (GNN) approaches can only maintain permutation-invariance, failing to guarantee invariance with respect to other transformations. Besides GNNs, other works design sophisticated transformation-invariant layers, which are computationally expensive and difficult to be extended. To solve this problem, we revisit why the existing neural networks cannot maintain transformation invariance when handling geometric data. Our findings show that transformation-invariant and distance-preserving initial representations are sufficient to achieve transformation invariance rather than needing sophisticated neural layer designs. Motivated by these findings, we propose Transformation Invariant Neural Networks (TinvNN), a straightforward and general framework for geometric data. Specifically, we realize transformation-invariant and distance-preserving initial point representations by modifying multi-dimensional scaling before feeding the representations into neural networks. We prove that TinvNN can strictly guarantee transformation invariance, being general and flexible enough to be combined with the existing neural networks. Extensive experimental results on point cloud analysis and combinatorial optimization demonstrate the effectiveness and general applicability of our proposed method. Based on the experimental results, we advocate that TinvNN should be considered a new starting point and an essential baseline for further studies of transformation-invariant geometric deep learning.

Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture

Does the Geometry of the Data Control the Geometry of Neural Predictions? (Student Abstract)

Training the Untrainable: Introducing Inductive Bias via Representational Alignment

Curved Geometric Networks for Visual Anomaly Recognition

Revisiting Transformation Invariant Geometric Deep Learning: Are Initial Representations All You Need?

InBiaseD: Inductive Bias Distillation to Improve Generalization and Robustness through Shape-awareness

Human-Like Geometric Abstraction in Large Pre-trained Neural Networks

Rethink Deep Learning with Invariance in Data Representation

Geometry-Informed Neural Networks

Relational Constraints On Neural Networks Reproduce Human Biases towards Abstract Geometric Regularity

A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks

Inductive Bias of Deep Convolutional Networks through Pooling Geometry

Tilt your Head: Activating the Hidden Spatial-Invariance of Classifiers

Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks

Neural networks learn to magnify areas near decision boundaries

A Relational Inductive Bias for Dimensional Abstraction in Neural Networks

On Inductive Biases That Enable Generalization of Diffusion Transformers

The Geometric Occam's Razor Implicit in Deep Learning

Unveiling and Mitigating Generalized Biases of DNNs through the Intrinsic Dimensions of Perceptual Manifolds

On the Geometry of Deep Learning

Theoretical Analysis of Inductive Biases in Deep Convolutional Networks