Abstract:The internal representations of early deep artificial neural networks (ANNs) were found to be remarkably similar to the internal neural representations measured experimentally in the primate brain. Here we ask, as deep ANNs have continued to evolve, are they becoming more or less brain-like? ANNs that are most functionally similar to the brain will contain mechanisms that are most like those used by the brain. We therefore developed Brain-Score – a composite of multiple neural and behavioral benchmarks that score any ANN on how similar it is to the brain’s mechanisms for core object recognition – and we deployed it to evaluate a wide range of state-of-the-art deep ANNs. Using this scoring system, we here report that: (1) DenseNet-169, CORnet-S and ResNet-101 are the most brain-like ANNs. (2) There remains considerable variability in neural and behavioral responses that is not predicted by any ANN, suggesting that no ANN model has yet captured all the relevant mechanisms. (3) Extending prior work, we found that gains in ANN ImageNet performance led to gains on Brain-Score. However, correlation weakened at ≥ 70% top-1 ImageNet performance, suggesting that additional guidance from neuroscience is needed to make further advances in capturing brain mechanisms. (4) We uncovered smaller (i.e. less complex) ANNs that are more brain-like than many of the best-performing ImageNet models, which suggests the opportunity to simplify ANNs to better understand the ventral stream. The scoring system used here is far from complete. However, we propose that evaluating and tracking model-benchmark correspondences through a Brain-Score that is regularly updated with new brain data is an exciting opportunity: experimental benchmarks can be used to guide machine network evolution, and machine networks are mechanistic hypotheses of the brain’s network and thus drive next experiments. To facilitate both of these, we release Brain-Score.org : a platform that hosts the neural and behavioral benchmarks, where ANNs for visual processing can be submitted to receive a Brain-Score and their rank relative to other models, and where new experimental data can be naturally incorporated.

The Neural Representation Benchmark and its Evaluation on Brain and Machine

Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition

Comparison Against Task Driven Artificial Neural Networks Reveals Functional Organization of Mouse Visual Cortex

Probing neural representations of scene perception in a hippocampally dependent task using artificial neural networks

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

Achieving More Human Brain-Like Vision via Human EEG Representational Alignment

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex

A Neurobiological Evaluation Metric for Neural Network Model Search

Evaluating (and Improving) the Correspondence Between Deep Neural Networks and Human Representations

Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?

Brain-inspired Models for Visual Object Recognition: an Overview

Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos

MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction

Exploration and Comparison of Deep Learning Architectures to Predict Brain Response to Realistic Pictures

Learning From Brains How to Regularize Machines

Improved object recognition using neural networks trained to mimic the brain's statistical properties

ReSi: A Comprehensive Benchmark for Representational Similarity Measures

Explaining face representation in the primate brain using different computational models