Aligned with LLM: a new multi-modal training paradigm for encoding fMRI activity in visual cortex

Shuxiao Ma,Linyuan Wang,Senbao Hou,Bin Yan

2024-01-08

Abstract:Recently, there has been a surge in the popularity of pre trained large language models (LLMs) (such as GPT-4), sweeping across the entire Natural Language Processing (NLP) and Computer Vision (CV) communities. These LLMs have demonstrated advanced multi-modal understanding capabilities and showcased strong performance across various benchmarks. The LLM has started to embody traits of artificial general intelligence, which holds vital guidance for enhancing brain-like characteristics within visual encoding models. Hence, This paper proposes a new multi-modal training paradigm, aligning with LLM, for encoding fMRI activity in visual cortex. Based on this paradigm, we trained an encoding model in fMRI data named the LLM-Visual Encoding Model (LLM-VEM). Specifically, we utilize LLM (miniGPT4) to generate descriptive text for all stimulus images, forming a high-quality textual description set. Moreover, we use the pre-trained text encoder (CLIP) to process these detailed descriptions, obtaining the text embedding features. Next, we use the contrast loss function to minimize the distance between the image embedding features and the text embedding features to complete the alignment operation of the stimulus image and text information. With the assistance of the pre-trained LLM, this alignment process facilitates better learning of the visual encoding model, resulting in higher precision. The final experimental results indicate that our training paradigm has significantly aided in enhancing the performance of the visual encoding model.

Computer Vision and Pattern Recognition,Neurons and Cognition

What problem does this paper attempt to address?

The paper proposes a new multimodal training paradigm that aligns with large-scale language models (LLMs) for encoding functional magnetic resonance imaging (fMRI) activity in the visual cortex. In the study, the authors utilize LLMs such as GPT-4 to generate descriptive text for stimulus images, creating a high-quality text description set. Then, these descriptions are processed using a pretrained text encoder like CLIP to obtain text embedding features. By minimizing the distance between image embedding features and text embedding features using a contrastive loss function, the alignment of image and text information is achieved. This approach helps the visual encoding model to learn better and improves prediction accuracy. Experimental results demonstrate that this training paradigm significantly enhances the performance of the visual encoding model.