Abstract:Recently, leveraging big data in deep learning has led to significant performance improvements, as confirmed in applications like mental state decoding using fMRI data. However, fMRI datasets remain relatively small in scale, and the inherent issue of low signal-to-noise ratios (SNR) in fMRI data further exacerbates these challenges. To address this, we apply compression techniques as a preprocessing step for fMRI data. We propose BrainCodec, a novel fMRI codec inspired by the neural audio codec. We evaluated BrainCodec's compression capability in mental state decoding, demonstrating further improvements over previous methods. Furthermore, we analyzed the latent representations obtained through BrainCodec, elucidating the similarities and differences between task and resting state fMRI, highlighting the interpretability of BrainCodec. Additionally, we demonstrated that fMRI reconstructions using BrainCodec can enhance the visibility of brain activity by achieving higher SNR, suggesting its potential as a novel denoising method. Our study shows that BrainCodec not only enhances performance over previous methods but also offers new analytical possibilities for neuroscience. Our codes, dataset, and model weights are available at <a class="link-external link-https" href="https://github.com/amano-k-lab/BrainCodec" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to solve several key challenges in functional magnetic resonance imaging (fMRI) data in neuroscience research: 1. **Small - scale data**: Although deep learning has achieved significant performance improvements on large - scale datasets, the scale of fMRI datasets is relatively small, which limits the training effect of the model. 2. **Low signal - to - noise ratio (SNR)**: The inherent low signal - to - noise ratio problem in fMRI data further exacerbates the above challenges, making it more difficult to extract useful information from these data. 3. **Insufficient pre - processing methods**: Existing pre - processing methods have limitations in improving the quality of fMRI data, especially in compression and denoising. To solve these problems, the authors propose a novel fMRI codec model named **BrainCodec**. Inspired by the neural audio codec (NAC), this model realizes effective compression and denoising of fMRI data by introducing the residual vector quantization (RVQ) technique. Specifically, the main contributions of BrainCodec include: - **Compression ability**: BrainCodec can effectively compress fMRI data, improve the signal - to - noise ratio, and thus improve the performance of downstream tasks (such as cognitive state decoding). - **Interpretability**: Through the analysis of the latent representation (i.e., the codebook), BrainCodec can reveal the similarities and differences between task - state and resting - state fMRI data, providing higher interpretability. - **Denoising ability**: The reconstruction results of BrainCodec can enhance the visualization of brain activity, indicating its potential as a new denoising method. ### Experimental results The paper verifies the effectiveness of BrainCodec through a series of experiments: 1. **Performance of downstream tasks**: - **Linear model**: After combining with BrainCodec, the performance of the linear model on the HCP and Over100 datasets is improved by approximately 31% and 23% respectively. - **CSM model**: After combining with BrainCodec, the performance of the CSM model on the HCP and Over100 datasets is also improved, especially performing well in the 100 - class classification task. 2. **Codebook analysis**: - **Comparison between task - state and resting - state**: Through UMAP visualization, it is found that the task - state codebook and the resting - state codebook form different clusters, verifying the uniqueness of the task - state fMRI data. - **Performance of partial codebooks**: Even if only part of the codebooks are used, the model performance still remains at a high level, indicating that the information learned by BrainCodec is highly effective and hierarchical. ### Conclusion BrainCodec not only outperforms existing methods in performance, but also provides new analysis possibilities, especially in interpreting the latent representation of fMRI data. This research provides a powerful tool for fMRI data analysis in the neuroscience field.

BrainCodec: Neural fMRI codec for the decoding of cognitive brain states

High-fidelity compression of extracellular recordings from motor cortex

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

SpatialCodec: Neural Spatial Speech Coding

A Fast Predictive Lossless Coder for Fmri Data Sets.

A Hybrid Autoencoder Framework of Dimensionality Reduction for Brain-Computer Interface Decoding

Deep learning of fMRI big data: a novel approach to subject-transfer decoding

SuperCodec: A Neural Speech Codec with Selective Back-Projection Network

AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec

FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec

Decoding brain states using functional magnetic resonance imaging

Group-level Brain Decoding with Deep Learning

An Adaptively Weighted Averaging Method for Regional Time Series Extraction of fMRI-based Brain Decoding

NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction

Functional annotation of human cognitive states using deep graph convolution

A Hybrid Spatio-Temporal Deep Belief Network and Sparse Representation-Based Framework Reveals Multi-Level Core Functional Components in Decoding Multi-Task fMRI Signals

A Compact Implicit Neural Representation for Efficient Storage of Massive 4D Functional Magnetic Resonance Imaging

Correntropy-Based Logistic Regression with Automatic Relevance Determination for Robust Sparse Brain Activity Decoding

Decoding the encoding of functional brain networks: An fMRI classification comparison of non-negative matrix factorization (NMF), independent component analysis (ICA), and sparse coding algorithms

MDCTCodec: A Lightweight MDCT-based Neural Audio Codec towards High Sampling Rate and Low Bitrate Scenarios