Abstract:Brain-related research topics in artificial intelligence have recently gained popularity, particularly due to the expansion of what multimodal architectures can do from computer vision to natural language processing. Our main goal in this work is to explore the possibilities and limitations of these architectures in spoken text decoding from non-invasive fMRI recordings. Contrary to vision and textual data, fMRI data represent a complex modality due to the variety of brain scanners, which implies (i) the variety of the recorded signal formats, (ii) the low resolution and noise of the raw signals, and (iii) the scarcity of pretrained models that can be leveraged as foundation models for generative learning. These points make the problem of the non-invasive decoding of text from fMRI recordings very challenging. In this paper, we propose and end-to-end multimodal LLM for decoding spoken text from fMRI signals. The proposed architecture is founded on (i) an encoder derived from a specific transformer incorporating an augmented embedding layer for the encoder and a better-adjusted attention mechanism than that present in the state of the art, and (ii) a frozen large language model adapted to align the embedding of the input text and the encoded embedding of brain activity to decode the output text. A benchmark in performed on a corpus consisting of a set of interactions human-human and human-robot interactions where fMRI and conversational signals are recorded synchronously. The obtained results are very promising, as our proposal outperforms the evaluated models, and is able to generate text capturing more accurate semantics present in the ground truth. The implementation code is provided in <a class="link-external link-https" href="https://github.com/Hmamouche/brain_decode" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to explore the capabilities and limitations of multimodal large language models (Multimodal LLM) in decoding spoken text from non-invasive functional magnetic resonance imaging (fMRI) recordings. Specifically, the paper focuses on the following points: 1. **Handling Complex Modalities**: Unlike data in computer vision and natural language processing, fMRI data is characterized by multiple formats, low resolution, and high noise, and lacks pre-trained foundational models. These factors make non-invasive text decoding from fMRI recordings highly challenging. 2. **Accuracy of Semantic Reconstruction**: Existing methods can generate text but still fall short in capturing the true semantics of the original stimuli. The paper proposes a new end-to-end multimodal LLM aimed at improving the semantic accuracy of decoded text. 3. **Potential for Practical Applications**: The technology for decoding text from fMRI recordings has broad application prospects in assistive communication technologies, brain-computer interfaces, and potential mind-reading technologies. These applications can improve direct communication between humans and external devices and could even be used to treat cognitive disorders such as autism, PTSD, and locked-in syndrome. ### Main Contributions 1. **Improved Transformer Encoder**: The paper proposes an improved Transformer encoder that better handles fMRI data through enhanced embedding layers and adjusted attention mechanisms. 2. **Integration of Multimodal LLM**: The paper combines the improved Transformer encoder with a frozen large language model (LLM), achieving understanding and support for multimodal representations through embedding alignment techniques. 3. **Experimental Validation**: The paper benchmarks the proposed model on a dataset containing synchronized fMRI and dialogue signals from human-human and human-robot interactions. The results show that the proposed model outperforms existing models in various text similarity and semantic metrics, generating more accurate semantic content. ### Conclusion By proposing a new multimodal LLM architecture, this paper successfully improves the accuracy and robustness of decoding spoken text from fMRI recordings, providing important references for future research and practical applications.

A multimodal LLM for the non-invasive decoding of spoken text from brain recordings

Speech neuromuscular decoding based on spectrogram images using conformal predictors with Bi-LSTM.

Decoding speech from non-invasive brain recordings

Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models

Decoding speech perception from non-invasive brain recordings

MindSpeech: Continuous Imagined Speech Decoding using High-Density fNIRS and Prompt Tuning for Advanced Human-AI Interaction

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

Modality-Agnostic fMRI Decoding of Vision and Language

Open-vocabulary Auditory Neural Decoding Using fMRI-prompted LLM

NeuSpeech: Decode Neural signal as Speech

Artificial Intelligence Based Multimodal Language Decoding from Brain Activity: A Review

MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model

LLM4Brain: Training a Large Language Model for Brain Video Understanding

Decoding Continuous Character-based Language from Non-invasive Brain Recordings

Brain Dialogue Interface (BDI): A User-Friendly fMRI Model for Interactive Brain Decoding

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

Decoding individual words from non-invasive brain recordings across 723 participants

Towards Unified Neural Decoding of Perceived, Spoken and Imagined Speech from EEG Signals

Efficient Neural Decoding Based on Multimodal Training

Interpretable prediction of brain activity during conversations from multimodal behavioral signals