Abstract:BACKGROUND AND OBJECTIVE: Most studies used neural activities evoked by linguistic stimuli such as phrases or sentences to decode the language structure. However, compared to linguistic stimuli, it is more common for the human brain to perceive the outside world through non-linguistic stimuli such as natural images, so only relying on linguistic stimuli cannot fully understand the information perceived by the human brain. To address this, an end-to-end mapping model between visual neural activities evoked by non-linguistic stimuli and visual contents is demanded.METHODS: Inspired by the success of the Transformer network in neural machine translation and the convolutional neural network (CNN) in computer vision, here a CNN-Transformer hybrid language decoding model is constructed in an end-to-end fashion to decode functional magnetic resonance imaging (fMRI) signals evoked by natural images into descriptive texts about the visual stimuli. Specifically, this model first encodes a semantic sequence extracted by a two-layer 1D CNN from the multi-time visual neural activity into a multi-level abstract representation, then decodes this representation, step by step, into an English sentence.RESULTS: Experimental results show that the decoded texts are semantically consistent with the corresponding ground truth annotations. Additionally, by varying the encoding and decoding layers and modifying the original positional encoding of the Transformer, we found that a specific architecture of the Transformer is required in this work.CONCLUSIONS: The study results indicate that the proposed model can decode the visual neural activities evoked by natural images into descriptive text about the visual stimuli in the form of sentences. Hence, it may be considered as a potential computer-aided tool for neuroscientists to understand the neural mechanism of visual information processing in the human brain in the future.

CNNs and Transformers Perceive Hybrid Images Similar to Humans

A Novel Transformer Network with a CNN-Enhanced Cross-Attention Mechanism for Hyperspectral Image Classification

A hybrid convolution transformer for hyperspectral image classification

A CNN-transformer hybrid approach for decoding visual neural activity into text

Do Vision Transformers See Like Convolutional Neural Networks?

A survey of the Vision Transformers and their CNN-Transformer based Variants

Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation

CMT: Convolutional Neural Networks Meet Vision Transformers

Image harmonization with Simple Hybrid CNN-Transformer Network

Adaptive Split-Fusion Transformer

Convolutional Embedding Makes Hierarchical Vision Transformer Stronger

Advancing AI-Generated Image Detection: Enhanced Accuracy through CNN and Vision Transformer Models with Explainable AI Insights

Transform-Invariant Convolutional Neural Networks for Image Classification and Search

Partial success in closing the gap between human and machine vision

Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A survey

The human visual system and CNNs can both support robust online translation tolerance following extreme displacements

Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets

RNNs, CNNs and Transformers in Human Action Recognition: A Survey and a Hybrid Model

Taming Transformers for High-Resolution Image Synthesis

Demystify Transformers & Convolutions in Modern Image Deep Networks

Dissociable Neural Representations of Adversarially Perturbed Images in Convolutional Neural Networks and the Human Brain