Abstract:Extensive literature has drawn comparisons between recordings of biological neurons in the brain and deep neural networks. This comparative analysis aims to advance and interpret deep neural networks and enhance our understanding of biological neural systems. However, previous works did not consider the time aspect and how the encoding of video and dynamics in deep networks relate to the biological neural systems within a large-scale comparison. Towards this end, we propose the first large-scale study focused on comparing video understanding models with respect to the visual cortex recordings using video stimuli. The study encompasses more than two million regression fits, examining image vs. video understanding, convolutional vs. transformer-based and fully vs. self-supervised models. Additionally, we propose a novel neural encoding scheme to better encode biological neural systems. We provide key insights on how video understanding models predict visual cortex responses; showing video understanding better than image understanding models, convolutional models are better in the early-mid visual cortical regions than transformer based ones except for multiscale transformers, and that two-stream models are better than single stream. Furthermore, we propose a novel neural encoding scheme that is built on top of the best performing video understanding models, while incorporating inter-intra region connectivity across the visual cortex. Our neural encoding leverages the encoded dynamics from video stimuli, through utilizing two-stream networks and multiscale transformers, while taking connectivity priors into consideration. Our results show that merging both intra and inter-region connectivity priors increases the encoding performance over each one of them standalone or no connectivity priors. It also shows the necessity for encoding dynamics to fully benefit from such connectivity priors.

One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment

Achieving More Human Brain-Like Vision via Human EEG Representational Alignment

Evaluating alignment between humans and neural network representations in image-based learning tasks

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

Aligning brain functions boosts the decoding of visual semantics in novel subjects

Complex Properties of Training Stimuli Affect Brain Alignment in a Deep Network Model of Mouse Visual Cortex

The Neural Representation Benchmark and its Evaluation on Brain and Machine

Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream

Exploring the Brain-like Properties of Deep Neural Networks: A Neural Encoding Perspective

Teaching CORnet Human fMRI Representations for Enhanced Model-Brain Alignment

Human Visual Pathways for Action Recognition versus Deep Convolutional Neural Networks: Representation Correspondence in Late but Not Early Layers

Optimizing deep video representation to match brain activity

Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)

How Aligned are Different Alignment Metrics?

AlignedCut: Visual Concepts Discovery on Brain-Guided Universal Feature Space

Aligning Machine and Human Visual Representations across Abstraction Levels

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

Dynamics Based Neural Encoding with Inter-Intra Region Connectivity

Can Brain Signals Reveal Inner Alignment with Human Languages?

Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks

Neural Dynamics of Object Manifold Alignment in the Ventral Stream