Abstract:Predicting the neural response to natural images in the visual cortex requires extracting relevant features from the images and relating those feature to the observed responses. In this work, we optimize the feature extraction in order to maximize the information shared between the image features and the neural response across voxels in a given region of interest (ROI) extracted from the BOLD signal measured by fMRI. We adapt contrastive learning (CL) to fine-tune a convolutional neural network, which was pretrained for image classification, such that a mapping of a given image's features are more similar to the corresponding fMRI response than to the responses to other images. We exploit the recently released Natural Scenes Dataset (Allen et al., 2022) as organized for the Algonauts Project (Gifford et al., 2023), which contains the high-resolution fMRI responses of eight subjects to tens of thousands of naturalistic images. We show that CL fine-tuning creates feature extraction models that enable higher encoding accuracy in early visual ROIs as compared to both the pretrained network and a baseline approach that uses a regression loss at the output of the network to tune it for fMRI response encoding. We investigate inter-subject transfer of the CL fine-tuned models, including subjects from another, lower-resolution dataset (Gong et al., 2023). We also pool subjects for fine-tuning to further improve the encoding performance. Finally, we examine the performance of the fine-tuned models on common image classification tasks, explore the landscape of ROI-specific models by applying dimensionality reduction on the Bhattacharya dissimilarity matrix created using the predictions on those tasks (Mao et al., 2024), and investigate lateralization of the processing for early visual ROIs using salience maps of the classifiers built on the CL-tuned models.

ACF: Aligned Contrastive Finetuning For Language and Vision Tasks

Label Anchored Contrastive Learning for Language Understanding

Adversarial Contrastive Learning via Asymmetric InfoNCE.

An Efficient Method of Supervised Contrastive Learning for Natural Language Understanding

Fine-grained Adaptive Contrastive Learning for Unsupervised Feature Extraction

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

CAVL: Learning Contrastive and Adaptive Representations of Vision and Language

Modeling Inner- and Cross-Task Contrastive Relations for Continual Image Classification

Adversarial Supervised Contrastive Learning

CLAF: Contrastive Learning with Augmented Features for Imbalanced Semi-Supervised Learning

Decoupled Contrastive Learning for Long-Tailed Recognition

SCLAV: Supervised Cross-modal Contrastive Learning for Audio-Visual Coding

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

Finetuning CLIP to Reason about Pairwise Differences

Model-Aware Contrastive Learning: Towards Escaping the Dilemmas

CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for Optimized Learning Fusion

Alignment Enhancement Network for Fine-grained Visual Categorization

Adaptive Contrastive Learning for Learning Robust Representations under Label Noise.

Adaptive Multi-head Contrastive Learning

An Asymmetric Contrastive Loss for Handling Imbalanced Datasets

Contrastive Learning to Fine-Tune Feature Extraction Models for the Visual Cortex