Abstract:Unit activity in particular deep neural networks (DNNs) are remarkably similar to the neuronal population responses to static images along the primate ventral visual cortex. Linear combinations of DNN unit activities are widely used to build predictive models of neuronal activity in the visual cortex. Nevertheless, prediction performance in these models is often investigated on stimulus sets consisting of everyday objects under naturalistic settings. Recent work has revealed a generalization gap in how predicting neuronal responses to synthetically generated out-of-distribution (OOD) stimuli. Here, we investigated how the recent progress in improving DNNs' object recognition generalization, as well as various DNN design choices such as architecture, learning algorithm, and datasets have impacted the generalization gap in neural predictivity. We came to a surprising conclusion that the performance on none of the common computer vision OOD object recognition benchmarks is predictive of OOD neural predictivity performance. Furthermore, we found that adversarially robust models often yield substantially higher generalization in neural predictivity, although the degree of robustness itself was not predictive of neural predictivity score. These results suggest that improving object recognition behavior on current benchmarks alone may not lead to more general models of neurons in the primate ventral visual cortex. Inspired by the neural circuits of the brain, deep neural networks (DNN) have been steadily improving in their ability to perform foundational visual tasks such as object recognition. Whereas, early models struggled with generalization to abstract visual domains such as line drawings and cartoons, recent advancement have approached near-human recognition capabilities. Moreover, the unit activity in these networks exhibit strong similarities with the activity of single-unit recordings along the primate ventral visual cortex. This capability of DNNs has provided visual neuroscientists with precise models for exploring the neural underpinnings of object recognition. Our research probes whether enhancements in neural networks' recognition of out-of-distribution objects correlate with improved predictability of brain activity in the visual cortex of monkeys to synthetic stimuli. We found that the out of distribution object recognition performance on natural image datasets is not a reliable measure of neural predictivity. However, DNN models that were trained to be more resilient to adversarially generated noise patterns as well as DNN ensembles, consistently yielded better generalization in neural predictivity. Altogether, our results suggest that improving object recognition behaviour on current benchmarks alone may not lead to more general models of neurons in the primate ventral visual cortex.

NeuralOOD: Improving Out-of-Distribution Generalization Performance with Brain-machine Fusion Learning Framework

Multimodal Contrastive Learning for Brain-Machine Fusion: from Brain-in-the-loop Modeling to Brain-out-of-the-loop Application

Two-Dimensional Attentive Fusion for Multi-Modal Learning of Neuroimaging and Genomics Data

Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex

Determinantal Point Process Attention Over Grid Cell Code Supports Out of Distribution Generalization

Multi-Net Fusion: Exploring a Brain-Inspired Neural Network Model for Facial Expression Recognition

A Context-Supported Deep Learning Framework for Multimodal Brain Imaging Classification

Brain-inspired Multimodal Learning Based on Neural Networks

Multiscale Brain-Like Neural Network for Saliency Prediction on Omnidirectional Images

How well do models of visual cortex generalize to out of distribution samples?

Densely Feature Fusion Based On Convolutional Neural Networks For Motor Imagery Eeg Classification

ADFCNN: Attention-Based Dual-Scale Fusion Convolutional Neural Network for Motor Imagery Brain-Computer Interface

Interpretable Multimodal Fusion Networks Reveal Mechanisms of Brain Cognition

A Heterogeneous Graph Based Framework for Multimodal Neuroimaging Fusion Learning

Distilling Multi-Scale Neural Mechanisms from Diverse Unlabeled Experimental Data Using Deep Domain-Adaptive Inference Framework

Multiscale Spatial-Temporal Feature Fusion Neural Network for Motor Imagery Brain-Computer Interfaces

Bayesian Cross-Modal Alignment Learning for Few-Shot Out-of-Distribution Generalization.

Data Augmentation for Motor Imagery Signal Classification Based on a Hybrid Neural Network.

Towards Robust Out-of-Distribution Generalization: Data Augmentation and Neural Architecture Search Approaches

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

Visual Image Decoding of Brain Activities Using a Dual Attention Hierarchical Latent Generative Network with Multiscale Feature Fusion