Abstract:Artificial and biological agents cannon learn given completely random and unstructured data. The structure of data is encoded in the metric relationships between data points. In the context of neural networks, neuronal activity within a layer forms a representation reflecting the transformation that the layer implements on its inputs. In order to utilize the structure in the data in a truthful manner, such representations should reflect the input distances and thus be continuous and isometric. Supporting this statement, recent findings in neuroscience propose that generalization and robustness are tied to neural representations being continuously differentiable. In machine learning, most algorithms lack robustness and are generally thought to rely on aspects of the data that differ from those that humans use, as is commonly seen in adversarial attacks. During cross-entropy classification, the metric and structural properties of network representations are usually broken both between and within classes. This side effect from training can lead to instabilities under perturbations near locations where such structure is not preserved. One of the standard solutions to obtain robustness is to add ad hoc regularization terms, but to our knowledge, forcing representations to preserve the metric structure of the input data as a stabilising mechanism has not yet been studied. In this work, we train neural networks to perform classification while simultaneously maintaining within-class metric structure, leading to isometric within-class representations. Such network representations turn out to be beneficial for accurate and robust inference. By stacking layers with this property we create a network architecture that facilitates hierarchical manipulation of internal neural representations. Finally, we verify that isometric regularization improves the robustness to adversarial attacks on MNIST.

Improved Generalization of Neural Classifiers with Enforced Internal Representation

Research on the Generalization Problem of BP Neural Network

Output representation of BP neural networks for classification problems

Representations and generalization in artificial and brain neural networks

Generalize Deep Neural Networks with Adaptive Regularization for Classifying

Generalization and Estimation Error Bounds for Model-based Neural Networks

Get the Best of Both Worlds: Improving Accuracy and Transferability by Grassmann Class Representation

Information-Theoretic Generalization Bounds for Deep Neural Networks

A new regularization learning method for improving generalization capability of neural network

State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations

Learning Internal Representations (COLT 1995)

Isometric Representations in Neural Networks Improve Robustness

Neuron with Steady Response Leads to Better Generalization

Remote Sensing Image Classification Based on Improved BP Neural Network

Representation Based Complexity Measures for Predicting Generalization in Deep Learning

On Generalization Bounds for Neural Networks with Low Rank Layers

Guided Random Projection: A Lightweight Feature Representation for Image Classification

Application of Improved BP Neural Network in Object Recognition

Formation of Representations in Neural Networks

Gradient-based inference of abstract task representations for generalization in neural networks

Generalization and Expressivity for Deep Nets