Abstract:The development of robust and generalisable models for encoding the spatio-temporal dynamics of human brain activity is crucial for advancing neuroscientific discoveries. However, significant individual variation in the organisation of the human cerebral cortex makes it difficult to identify population-level trends in these signals. Recently, Surface Vision Transformers (SiTs) have emerged as a promising approach for modelling cortical signals, yet they face some limitations in low-data scenarios due to the lack of inductive biases in their architecture. To address these challenges, this paper proposes the surface Masked AutoEncoder (sMAE) and video surface Masked AutoEncoder (vsMAE) - for multivariate and spatio-temporal pre-training of cortical signals over regular icosahedral grids. These models are trained to reconstruct cortical feature maps from masked versions of the input by learning strong latent representations of cortical structure and function. Such representations translate into better modelling of individual phenotypes and enhanced performance in downstream tasks. The proposed approach was evaluated on cortical phenotype regression using data from the young adult Human Connectome Project (HCP) and developing HCP (dHCP). Results show that (v)sMAE pre-trained models improve phenotyping prediction performance on multiple tasks by $\ge 26\%$, and offer faster convergence relative to models trained from scratch. Finally, we show that pre-training vision transformers on large datasets, such as the UK Biobank (UKB), supports transfer learning to low-data regimes. Our code and pre-trained models are publicly available at <a class="link-external link-https" href="https://github.com/metrics-lab/surface-masked-autoencoders" rel="external noopener nofollow">this https URL</a> .

The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision

Interpreting Attention Layer Outputs with Sparse Autoencoders

Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders

Sparse-Coding Variational Auto-Encoders

Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders

Sparse-Coding Variational Autoencoders

Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks

Disentangling Dense Embeddings with Sparse Autoencoders

Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs

Features that Make a Difference: Leveraging Gradients for Improved Dictionary Learning

Improving Dictionary Learning with Gated Sparse Autoencoders

Decomposing The Dark Matter of Sparse Autoencoders

A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

Sparse, Geometric Autoencoder Models of V1

Efficient Dictionary Learning with Switch Sparse Autoencoders

Spatio-Temporal Encoding of Brain Dynamics with Surface Masked Autoencoders

Transforming Auto-Encoders

Analyzing (In)Abilities of SAEs via Formal Languages

Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

HSAE: A Hessian Regularized Sparse Auto-Encoders.

Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders