Abstract:Synesthesia is a psychological phenomenon where sensory signals become mixed. Input to one sensory modality produces an experience in a second, unstimulated modality. In “grapheme-color synesthesia”, viewed letters and numbers evoke mental imagery of colors. The study of this condition has implications for increasing our understanding of brain architecture and function, language, memory and semantics, and the nature of consciousness. In this work, we propose a novel application of deep learning to model perception in grapheme-color synesthesia. Achromatic letter images, taken from database of handwritten characters, are used to train the model, and to induce computational synesthesia. Results show the model learns to accurately create a colored version of the inducing stimulus, according to a statistical distribution from experiments on a sample population of grapheme-color synesthetes. To the author’s knowledge, this work represents the first model that accurately produces spontaneous, creative mental imagery characteristic of the synesthetic perceptual experience. Experiments in cognitive science have contributed to our understanding of some of the observable behavioral effects of synesthesia, and previous models have outlined neural mechanisms that may account for these observations. A model of synesthesia that generates testable predictions on brain activity and behavior is needed to complement large scale data collection efforts in neuroscience, especially when articulating simple descriptions of cause (stimulus) and effect (behavior). The research and modeling approach reported here provides a framework that begins to address this need.

From pixels to notes: a computational implementation of synaesthesia for cultural artefacts

An Artistic Visualization of Music Modeling a Synesthetic Experience

Synesthesia AR: Creating Sound-to-Color Synesthesia in Augmented Reality

Image–Music Synesthesia-Aware Learning Based on Emotional Similarity Recognition

Visualization of isomorphism-synesthesia of colour and music

Synesthetic Perception: Alexander Scriabin's Color Hearing

Generation of Kandinsky Art

Robot Synesthesia: A Sound and Emotion Guided AI Painter

Synaesthesia, the arts and creativity: a neurological connection

Art of musical color: A synesthesia‐based mechanism of color art

A case of color–taste synesthesia

Audiovisual integration system based on synesthesia for people with impaired hearing

Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings

A Deep Learning Model of Perception in Color-Letter Synesthesia

A Unified Framework for Synaesthesia Analysis

Game design to determine the balance and difficulty of the game based on the synesthesia

Automaticity of pitch class-color synesthesia as revealed by a Stroop-like effect

Subtitled speech: Phenomenology of tickertape synesthesia

VisibleSound: Perceiving Environmental Sound with 4D Form

The color of smiling: computational synaesthesia of facial expressions

Music and Color Synaesthesia with Visual Saliency Information