Abstract:There is a growing interest in using generative adversarial networks (GANs) to produce image content that is indistinguishable from real images as judged by a typical person. A number of GAN variants for this purpose have been proposed; however, evaluating GAN performance is inherently difficult because current methods for measuring the quality of their output are not always consistent with what a human perceives. We propose a novel approach that combines a brain-computer interface (BCI) with GANs to generate a measure we call Neuroscore, which closely mirrors the behavioral ground truth measured from participants tasked with discerning real from synthetic images. This technique we call a neuro-AI interface, as it provides an interface between a human's neural systems and an AI process. In this paper, we first compare the three most widely used metrics in the literature for evaluating GANs in terms of visual quality and compare their outputs with human judgments. Secondly, we propose and demonstrate a novel approach using neural signals and rapid serial visual presentation (RSVP) that directly measures a human perceptual response to facial production quality, independent of a behavioral response measurement. The correlation between our proposed Neuroscore and human perceptual judgments has Pearson correlation statistics: r(48) = − 0.767, p = 2.089e − 10. We also present the bootstrap result for the correlation i.e., p ≤ 0.0001. Results show that our Neuroscore is more consistent with human judgment compared with the conventional metrics we evaluated. We conclude that neural signals have potential applications for high-quality, rapid evaluation of GANs in the context of visual image synthesis.

Cross-Subject EEG Feedback for Implicit Image Generation

Neurocognition-inspired Design with Machine Learning

Human-in-the-loop design with machine learning

MindCeive: Perceiving human imagination using CNN-GRU and GANs

NeuroGAN: image reconstruction from EEG signals via an attention-based GAN

EEG2IMAGE: Image Reconstruction from EEG Brain Signals

Use of Neural Signals to Evaluate the Quality of Generative Adversarial Network Performance in Facial Image Generation

Learning via social awareness: Improving a deep generative sketching model with facial feedback

Verifying Design through Generative Visualization of Neural Activities

BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance

NeuroGen: Activation optimized image synthesis for discovery neuroscience

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

Synthetic-Neuroscore: Using A Neuro-AI Interface for Evaluating Generative Adversarial Networks

Visual Saliency and Image Reconstruction from EEG Signals via an Effective Geometric Deep Network-Based Generative Adversarial Network

A Neuro-AI Interface for Evaluating Generative Adversarial Networks

Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences

Deep Cross-Modal Audio-Visual Generation

Modulating human brain responses via optimal natural image selection and synthetic image generation

Common Spatial Generative Adversarial Networks based EEG Data Augmentation for Cross-Subject Brain-Computer Interface

Multi-Class Image Generation from EEG Features with Conditional Generative Adversarial Networks

Learning a face space for experiments on human identity