Abstract:Synesthesia is a psychological phenomenon where sensory signals become mixed. Input to one sensory modality produces an experience in a second, unstimulated modality. In “grapheme-color synesthesia”, viewed letters and numbers evoke mental imagery of colors. The study of this condition has implications for increasing our understanding of brain architecture and function, language, memory and semantics, and the nature of consciousness. In this work, we propose a novel application of deep learning to model perception in grapheme-color synesthesia. Achromatic letter images, taken from database of handwritten characters, are used to train the model, and to induce computational synesthesia. Results show the model learns to accurately create a colored version of the inducing stimulus, according to a statistical distribution from experiments on a sample population of grapheme-color synesthetes. To the author’s knowledge, this work represents the first model that accurately produces spontaneous, creative mental imagery characteristic of the synesthetic perceptual experience. Experiments in cognitive science have contributed to our understanding of some of the observable behavioral effects of synesthesia, and previous models have outlined neural mechanisms that may account for these observations. A model of synesthesia that generates testable predictions on brain activity and behavior is needed to complement large scale data collection efforts in neuroscience, especially when articulating simple descriptions of cause (stimulus) and effect (behavior). The research and modeling approach reported here provides a framework that begins to address this need.

Synesthesia AR: Creating Sound-to-Color Synesthesia in Augmented Reality

VisibleSound: Perceiving Environmental Sound with 4D Form

Visualization of isomorphism-synesthesia of colour and music

Audiovisual integration system based on synesthesia for people with impaired hearing

An Artistic Visualization of Music Modeling a Synesthetic Experience

Art of musical color: A synesthesia‐based mechanism of color art

From pixels to notes: a computational implementation of synaesthesia for cultural artefacts

SonifyAR: Context-Aware Sound Generation in Augmented Reality

Synesthesia and the Color-light Keyboards

The Research on Digital Audio-Visual Synesthesia

The Application of Synesthetic Principles to Foster Musical Creativity

Computational Trichromacy Reconstruction: Empowering the Color-Vision Deficient to Recognize Colors Using Augmented Reality

Enhancing Art Gallery Visitors' Experiences through Audio Augmented Reality Technology

SonifEye: Sonification of Visual Information Using Physical Modeling Sound Synthesis

Musical pitch classes have rainbow hues in pitch class-color synesthesia

A Deep Learning Model of Perception in Color-Letter Synesthesia

Robot Synesthesia: A Sound and Emotion Guided AI Painter

Emotions and Consciousness Alterations in Music-color Synesthesia

SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness

Study on Application of Synesthesia using Smart Technology

Automaticity of pitch class-color synesthesia as revealed by a Stroop-like effect