Abstract:How do we know if two systems - biological or artificial - process information in a similar way? Similarity measures such as linear regression, Centered Kernel Alignment (CKA), Normalized Bures Similarity (NBS), and angular Procrustes distance, are often used to quantify this similarity. However, it is currently unclear what drives high similarity scores and even what constitutes a "good" score. Here, we introduce a novel tool to investigate these questions by differentiating through similarity measures to directly maximize the score. Surprisingly, we find that high similarity scores do not guarantee encoding task-relevant information in a manner consistent with neural data; and this is particularly acute for CKA and even some variations of cross-validated and regularized linear regression. We find no consistent threshold for a good similarity score - it depends on both the measure and the dataset. In addition, synthetic datasets optimized to maximize similarity scores initially learn the highest variance principal component of the target dataset, but some methods like angular Procrustes capture lower variance dimensions much earlier than methods like CKA. To shed light on this, we mathematically derive the sensitivity of CKA, angular Procrustes, and NBS to the variance of principal component dimensions, and explain the emphasis CKA places on high variance components. Finally, by jointly optimizing multiple similarity measures, we characterize their allowable ranges and reveal that some similarity measures are more constraining than others. While current measures offer a seemingly straightforward way to quantify the similarity between neural systems, our work underscores the need for careful interpretation. We hope the tools we developed will be used by practitioners to better understand current and future similarity measures.

Decoupling Semantic Similarity from Spatial Alignment for Neural Networks

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Semantic similarity metrics for image registration

A Hybrid Semantic Similarity Measurement for Geospatial Entities

Not just a matter of semantics: the relationship between visual similarity and semantic similarity

Measuring similarity between embedding spaces using induced neighborhood graphs

Learning similarity for semantic images classification

Exploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency

Addressing Discrepancies in Semantic and Visual Alignment in Neural Networks

Spatial Scene Similarity Assessment Based on Deep Learning

Dimensions underlying the representational alignment of deep neural networks with humans

ExplainLFS: Explaining neural architectures for similarity learning from local perturbations in the latent feature space

Latent Space Translation via Semantic Alignment

A high-throughput approach for the efficient prediction of perceived similarity of natural objects

Learning Non-Metric Visual Similarity for Image Retrieval

Similarity of Neural Network Models: A Survey of Functional and Representational Measures

Differentiable Optimization of Similarity Scores Between Models and Brains

What Representational Similarity Measures Imply about Decodable Information

Latent Space Translation via Inverse Relative Projection

Visualizing Deep Similarity Networks

Evaluation of taxonomic and neural embedding methods for calculating semantic similarity