Foundation model of neural activity predicts response to new stimulus types and anatomy

Eric Y Wang,Paul G. Fahey,Zhuokun Ding,Stelios Papadopoulos,Kayla Ponder,Mara Weis,Andersen Chang,Taliah Muhammad,Saumil Patel,Zhiwei Ding,Dat T. Tran,Jiakun Fu,Casey M. Schneider-Mizell,R. Clay Reid,Forrest Collman,Nuno Macarico da Costa,Katrin Franke,Alexander S. Ecker,Jacob Reimer,Xaq Pitkow,Fabian H. Sinz,Andreas S. Tolias

DOI: https://doi.org/10.1101/2023.03.21.533548

2024-08-28

Abstract:The complexity of neural circuits makes it challenging to decipher the brain's algorithms of intelligence. Recent breakthroughs in deep learning have produced models that accurately simulate brain activity, enhancing our understanding of the brain's computational objectives and neural coding. However, these models struggle to generalize beyond their training distribution, limiting their utility. The emergence of foundation models, trained on vast datasets, has introduced a new AI paradigm with remarkable generalization capabilities. We collected large amounts of neural activity from visual cortices of multiple mice and trained a foundation model to accurately predict neuronal responses to arbitrary natural videos. This model generalized to new mice with minimal training and successfully predicted responses across various new stimulus domains, such as coherent motion and noise patterns. It could also be adapted to new tasks beyond neural prediction, accurately predicting anatomical cell types, dendritic features, and neuronal connectivity within the MICrONS functional connectomics dataset. Our work is a crucial step toward building foundation brain models. As neuroscience accumulates larger, multi-modal datasets, foundation models will uncover statistical regularities, enabling rapid adaptation to new tasks and accelerating research.

Neuroscience