Abstract:Sensory substitution devices (SSDs) such as the 'vOICe' preserve visual information in sound by turning visual height, brightness, and laterality into auditory pitch, volume, and panning/time respectively. However, users have difficulty identifying or tracking multiple simultaneously presented tones - a skill necessary to discriminate the upper and lower edges of object shapes. We explore how these deficits can be addressed by using image-sonifications inspired by auditory scene analysis (ASA). Here, sighted subjects (N=25) of varying musical experience listened to, and then reconstructed, complex shapes consisting of simultaneously presented upper and lower lines. Complex shapes were sonified using the vOICe, with either the upper and lower lines varying only in pitch (i.e. the vOICe's 'unaltered' default settings), or with one line degraded to alter its auditory timbre or volume. Results found that overall performance increased with subjects' years of prior musical experience. ANOVAs revealed that both sonification style and musical experience significantly affected performance, but with no interaction effect between them. Compared to the vOICe's 'unaltered' pitch-height mapping, subjects had significantly better image-reconstruction abilities when the lower line was altered via timbre or volume-modulation. By contrast, altering the upper line only helped users identify the unaltered lower line. In conclusion, adding ASA principles to vision-to-audio SSDs boosts subjects' image-reconstruction abilities, even if this also reduces total task-relevant information. Future SSDs should seek to exploit these findings to enhance both novice user abilities and the use of SSDs as visual rehabilitation tools.

Audiovisual integration system based on synesthesia for people with impaired hearing

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

VisibleSound: Perceiving Environmental Sound with 4D Form

The Research on Digital Audio-Visual Synesthesia

Hybrid auditory feedback: a new method for mobility assistance of the visually impaired.

Synesthesia AR: Creating Sound-to-Color Synesthesia in Augmented Reality

Perceiving Graphical and Pictorial Information via Hearing and Touch

Cross-modal codification of images with auditory stimuli: a language for the visually impaired

Study on Application of Synesthesia using Smart Technology

Visualization of isomorphism-synesthesia of colour and music

Speech visualization by integrating features for the hearing impaired

Proximal auditory AR rehabilitation: system integration and wellness applications-from hearing support up toward vestibular rehabilitation

Highly compromised auditory spatial perception in aided congenitally hearing-impaired and rapid improvement with tactile technology

An Artistic Visualization of Music Modeling a Synesthetic Experience

Audio drawing and working memory in blindness: design, development and validation of a multi-sensory system

Optimality and Limitations of Audio-Visual Integration for Cognitive Systems

Colorophone 2.0: A Wearable Color Sonification Device Generating Live Stereo-Soundscapes-Design, Implementation, and Usability Audit

Auditory Scene Analysis Principles Improve Image Reconstruction Abilities of Novice Vision-to-Audio Sensory Substitution Users

Autoencoding sensory substitution

Synesthesia in a congenitally blind individual

Listen to the Image