Meenakshi Khosla,Alex H. Williams
Abstract:Common measures of neural representational (dis)similarity are designed to be insensitive to rotations and reflections of the neural activation space. Motivated by the premise that the tuning of individual units may be important, there has been recent interest in developing stricter notions of representational (dis)similarity that require neurons to be individually matched across networks. When two networks have the same size (i.e. same number of neurons), a distance metric can be formulated by optimizing over neuron index permutations to maximize tuning curve alignment. However, it is not clear how to generalize this metric to measure distances between networks with different sizes. Here, we leverage a connection to optimal transport theory to derive a natural generalization based on "soft" permutations. The resulting metric is symmetric, satisfies the triangle inequality, and can be interpreted as a Wasserstein distance between two empirical distributions. Further, our proposed metric avoids counter-intuitive outcomes suffered by alternative approaches, and captures complementary geometric insights into neural representations that are entirely missed by rotation-invariant metrics.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: How to measure the similarity of single - neuron tuning in neural networks, especially in the case of different network scales. Specifically, the paper proposes a new metric method - Soft Matching Distance - to overcome the limitations of existing methods when comparing neural networks of different sizes.
### Problem Background
Existing neural representation similarity measurement methods (such as Centered Kernel Alignment CKA, Canonical Correlation Analysis CCA, Procrustes distance, etc.) are usually insensitive to rotation and reflection, which means that they cannot capture the specific tuning characteristics of single neurons. Although some studies have begun to focus on methods that require one - to - one matching of neurons, these methods are only applicable to networks with the same number of neurons. When the number of neurons in two networks is different, how to effectively compare their neural representations becomes a challenge.
### Core Problem of the Paper
The main contribution of the paper lies in proposing a new metric method - Soft Matching Distance - which is based on optimal transport theory and can handle the comparison between neural networks of different sizes. Specifically:
1. **Soft Matching Distance**: By introducing "soft" permutation matrices, this method can perform effective matching between neural networks of different sizes. This metric method not only retains the spatial symmetry and triangle inequality properties of the original metric, but also can avoid counter - intuitive results that may occur in other methods.
2. **Geometric Insight**: The Soft Matching Distance can capture the geometric features of neural representations that are ignored by rotation - invariant metrics, thus providing a new tool for studying the importance of single - neuron tuning.
3. **Application Verification**: The paper demonstrates the effectiveness of the Soft Matching Distance in practical applications through experiments. For example, in deep convolutional neural networks (DNNs), it can reveal the non - arbitrariness of neural representations under different network architectures and training data sets, and also performs well in model - brain comparison studies.
### Formula Representation
The Soft Matching Distance is defined as follows:
\[
d_T(X, Y)=\sqrt{\min_{P \in T(N_x, N_y)} \sum_{i, j} P_{ij}\|x_i - y_j\|^2}
\]
where \(T(N_x, N_y)\) is the transportation polytope, defined as:
\[
T(N_x, N_y)=\left\{P \in \mathbb{R}^{N_x\times N_y}\mid\sum_i P_{ij}=\frac{1}{N_y}, \sum_j P_{ij}=\frac{1}{N_x}, P_{ij}\geq0\right\}
\]
In addition, the paper also proposes the Soft Matching Correlation Score:
\[
s_T(X, Y)=\max_{P \in T(N_x, N_y)} \sum_{i, j} P_{ij}x_i^T y_j
\]
These formulas enable the Soft Matching Distance to not only measure the distance between neural networks, but also quantify their similarity through the correlation score.
### Summary
In summary, this paper aims to solve the limitations of existing neural representation similarity measurement methods when dealing with neural networks of different sizes, and proposes a new metric method - Soft Matching Distance - which can more accurately capture the tuning characteristics of single neurons and shows superior performance in multiple application scenarios.