Abstract:A key requirement for the development of effective learning representations is their evaluation and comparison to representations we know to be effective. In natural sensory domains, the community has viewed the brain as a source of inspiration and as an implicit benchmark for success. However, it has not been possible to directly test representational learning algorithms directly against the representations contained in neural systems. Here, we propose a new benchmark for visual representations on which we have directly tested the neural representation in multiple visual cortical areas in macaque (utilizing data from [Majaj et al., 2012]), and on which any computer vision algorithm that produces a feature space can be tested. The benchmark measures the effectiveness of the neural or machine representation by computing the classification loss on the ordered eigendecomposition of a kernel matrix [Montavon et al., 2011]. In our analysis we find that the neural representation in visual area IT is superior to visual area V4. In our analysis of representational learning algorithms, we find that three-layer models approach the representational performance of V4 and the algorithm in [Le et al., 2012] surpasses the performance of V4. Impressively, we find that a recent supervised algorithm [Krizhevsky et al., 2012] achieves performance comparable to that of IT for an intermediate level of image variation difficulty, and surpasses IT at a higher difficulty level. We believe this result represents a major milestone: it is the first learning algorithm we have found that exceeds our current estimate of IT representation performance. We hope that this benchmark will assist the community in matching the representational performance of visual cortex and will serve as an initial rallying point for further correspondence between representations derived in brains and machines.

ReSi: A Comprehensive Benchmark for Representational Similarity Measures

Similarity of Neural Network Models: A Survey of Functional and Representational Measures

The Neural Representation Benchmark and its Evaluation on Brain and Machine

ReCo: A Modular Neural Framework for Automatically Recommending Connections in Software Models

NAS-Bench-101: Towards Reproducible Neural Architecture Search

Evaluating Representational Similarity Measures from the Lens of Functional Correspondence

Bayesian Comparisons Between Representations

ContraSim -- Analyzing Neural Representations Based on Contrastive Learning

Decoupling Semantic Similarity from Spatial Alignment for Neural Networks

Towards Measuring Representational Similarity of Large Language Models

Generalized Shape Metrics on Neural Representations

Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains

Benchmarking Meaning Representations in Neural Semantic Parsing

BigNeuron: A Resource to Benchmark and Predict Best-Performing Algorithms for Automated Reconstruction of Neuronal Morphology

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing

A Neuro-Symbolic Benchmark Suite for Concept Quality and Reasoning Shortcuts

Revisit and Benchmarking of Automated Quantization Toward Fair Comparison

RepMatch: Quantifying Cross-Instance Similarities in Representation Space

Similarity of Neural Network Representations Revisited

What Representational Similarity Measures Imply about Decodable Information

Neural Reconstruction Integrity: A metric for assessing the connectivity of reconstructed neural networks