EidetiCom: A Cross-modal Brain-Computer Semantic Communication Paradigm for Decoding Visual Perception

Linfeng Zheng,Peilin Chen,Shiqi Wang

2024-07-21

Abstract:Brain-computer interface (BCI) facilitates direct communication between the human brain and external systems by utilizing brain signals, eliminating the need for conventional communication methods such as speaking, writing, or typing. Nevertheless, the continuous generation of brain signals in BCI frameworks poses challenges for efficient storage and real-time transmission. While considering the human brain as a semantic source, the meaningful information associated with cognitive activities often gets obscured by substantial noise present in acquired brain signals, resulting in abundant redundancy. In this paper, we propose a cross-modal brain-computer semantic communication paradigm, named EidetiCom, for decoding visual perception under limited-bandwidth constraint. The framework consists of three hierarchical layers, each responsible for compressing the semantic information of brain signals into representative features. These low-dimensional compact features are transmitted and converted into semantically meaningful representations at the receiver side, serving three distinct tasks for decoding visual perception: brain signal-based visual classification, brain-to-caption translation, and brain-to-image generation, in a scalable manner. Through extensive qualitative and quantitative experiments, we demonstrate that the proposed paradigm facilitates the semantic communication under low bit rate conditions ranging from 0.017 to 0.192 bits-per-sample, achieving high-quality semantic reconstruction and highlighting its potential for efficient storage and real-time communication of brain recordings in BCI applications, such as eidetic memory storage and assistive communication for patients.

Multimedia

What problem does this paper attempt to address?

The paper aims to address the challenges of efficient storage and real-time transmission brought by continuous brain signal generation in the Brain-Computer Interface (BCI) framework. Specifically, the authors propose a cross-modal brain-machine semantic communication paradigm called EidetiCom, which is used to decode visual perception under bandwidth-limited conditions. This paradigm achieves its goals through the following methods:

1. **Compression of Brain Signals**: EidetiCom compresses the semantic information of brain signals into representative features. These low-dimensional compact features can be transmitted and converted into semantically meaningful representations at the receiving end.
2. **Multi-task Decoding**: EidetiCom supports three different tasks to decode visual perception:
- Visual classification based on brain signals
- Brain-to-text translation
- Brain-to-image generation
3. **Scalability**: The framework has a three-layer structure, with each layer responsible for compressing the semantic information in brain signals. These features can be utilized at different granularity levels to meet the needs of various application scenarios, such as memory storage or patient-assisted communication.

Through extensive qualitative and quantitative experiments, the authors demonstrate that EidetiCom can achieve high-quality semantic reconstruction under low bit-rate conditions (from 0.017 to 0.192 bits-per-sample) and highlight its potential for efficient storage and real-time communication in BCI applications.