Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models

Eleonora Lopez,Luigi Sigillo,Federica Colonnese,Massimo Panella,Danilo Comminiello
2024-09-18
Abstract:Generating images from brain waves is gaining increasing attention due to its potential to advance brain-computer interface (BCI) systems by understanding how brain signals encode visual cues. Most of the literature has focused on fMRI-to-Image tasks as fMRI is characterized by high spatial resolution. However, fMRI is an expensive neuroimaging modality and does not allow for real-time BCI. On the other hand, electroencephalography (EEG) is a low-cost, non-invasive, and portable neuroimaging technique, making it an attractive option for future real-time applications. Nevertheless, EEG presents inherent challenges due to its low spatial resolution and susceptibility to noise and artifacts, which makes generating images from EEG more difficult. In this paper, we address these problems with a streamlined framework based on the ControlNet adapter for conditioning a latent diffusion model (LDM) through EEG signals. We conduct experiments and ablation studies on popular benchmarks to demonstrate that the proposed method beats other state-of-the-art models. Unlike these methods, which often require extensive preprocessing, pretraining, different losses, and captioning models, our approach is efficient and straightforward, requiring only minimal preprocessing and a few components. Code will be available after publication.
Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
The problem this paper attempts to address is the challenge of generating images from electroencephalogram (EEG) signals. Specifically, the paper aims to efficiently generate high-quality and semantically correct images from EEG signals through a simplified approach using ControlNet adapters and Latent Diffusion Models (LDM). Compared to existing methods, this approach does not require complex preprocessing, pre-training, or additional loss functions, thereby achieving a simpler and more efficient brain-computer interface (BCI) system. Moreover, this method overcomes the inherent low spatial resolution and noise issues of EEG signals, making image generation from EEG more feasible, especially in real-time applications.