Abstract:Visual neural decoding aims to unlock the mysteries of how the human brain interprets the visual world. While early studies made some progress in decoding visual activity for singular type of information, they failed to concurrently reveal the multi-level interweaving linguistic information in the brain. Here, we developed a novel Visual Language Decoding Model (VLDM) capable of decoding categories, semantic labels, and textual descriptions from visual perceptual activities simultaneously. We selected the large-scale NSD dataset to ensure the efficiency of the decoding model in joint training and evaluation across multiple tasks. For category decoding, we achieved the effective classification of 12 categories with an accuracy of nearly 70%, significantly surpassing the chance level. For label decoding, we attained the precise prediction of 80 specific semantic labels with a 16-fold improvement over the chance level. For text decoding, the scores of the decoded text surpassed the corresponding baseline levels by remarkable margins on six evaluation metrics. This study contributes significantly to extensive applications in multi-layered brain-computer interfaces, potentially leading to more natural and efficient human-computer interaction experiences.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to simultaneously decode multi - level language information, including categories, semantic labels and text descriptions, from the visual perception activities of the human brain. Specifically, the research aims to develop a multi - task Visual Language Decoding Model (VLDM) that can efficiently decode this information from functional magnetic resonance imaging (fMRI) data. ### Research Background and Problem Statement 1. **The Relationship between Visual Cognition and Language Processing**: - There are complex interactions between visual signals and language processing in the brain. Visual signals are mainly processed by the visual cortex, while language processing involves Broca's area and Wernicke's area. Research shows that there is a potential interdependence between language understanding and visual information processing. 2. **Limitations of Existing Research**: - Although early research has made some progress in decoding a single type of visual information, it has failed to simultaneously reveal the multi - level intertwined semantic information in the brain. - Most research adopts a single - task decoding method, lacking knowledge sharing among different decoding tasks, which limits the generalization ability of visual decoding. 3. **Advantages of Multi - task Learning**: - In the fields of computer vision and natural language processing, multi - task learning has been proven to improve the joint processing efficiency of multiple tasks. For example, the YOLOP network can simultaneously process object detection, drivable area segmentation and lane detection, significantly improving performance. ### Research Objectives To overcome the above limitations, the author proposes a new multi - task Visual Language Decoding Model (VLDM), which has the following characteristics: - **Multi - task Decoding**: It can simultaneously decode the category, semantic label and text description of an image. - **High Precision**: It is trained and evaluated with a large - scale NSD dataset to ensure the efficiency and accuracy of the model on multiple tasks. - **Application Value**: It provides multiple application scenarios for brain - computer interfaces, such as controlling wheelchairs, operating robotic arms and restoring language functions. ### Main Contributions - **Classification Decoding**: It has achieved effective classification of 12 categories with an accuracy rate close to 70%, significantly exceeding the random level. - **Label Decoding**: It accurately predicts 80 specific semantic labels, and the accuracy rate is 16 times higher than the random level. - **Text Decoding**: The generated text is significantly better than the baseline level in six evaluation indicators. Through these improvements, this research not only deepens the understanding of visual cognition and language processing mechanisms but also lays the foundation for more natural and efficient brain - computer interface applications in the future.

From Sight to Insight: A Multi-task Approach with the Visual Language Decoding Model

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

‘When’ and ‘what’ did you see? A novel fMRI-based visual decoding framework

A dual‐channel language decoding from brain activity with progressive transfer training

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

MindGPT: Interpreting What You See with Non-invasive Brain Recordings

Long short‐term memory‐based neural decoding of object categories evoked by natural images

Multi-Semantic Decoding of Visual Perception with Graph Neural Networks

Modality-Agnostic fMRI Decoding of Vision and Language

Visual Neural Decoding via Improved Visual-EEG Semantic Consistency

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

A neural decoding algorithm that generates language from visual activity evoked by natural images

Robust Decoding of Rich Dynamical Visual Scenes With Retinal Spikes

Decoding Generic Visual Representations From Human Brain Activity using Machine Learning

Brain Dialogue Interface (BDI): A User-Friendly fMRI Model for Interactive Brain Decoding

Toward Generalizing Visual Brain Decoding to Unseen Subjects

Efficient Neural Decoding Based on Multimodal Training

CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding

Revealing Vision-Language Integration in the Brain with Multimodal Networks

Decoding dynamic visual scenes across the brain hierarchy