Abstract:People experience the world through multiple senses simultaneously, contributing to our sense of place. Prior quantitative geography studies have mostly emphasized human visual perceptions, neglecting human auditory perceptions at place due to the challenges in characterizing the acoustic environment vividly. Also, few studies have synthesized the two-dimensional (auditory and visual) perceptions in understanding human sense of place. To bridge these gaps, we propose a Soundscape-to-Image Diffusion model, a generative Artificial Intelligence (AI) model supported by Large Language Models (LLMs), aiming to visualize soundscapes through the generation of street view images. By creating audio-image pairs, acoustic environments are first represented as high-dimensional semantic audio vectors. Our proposed Soundscape-to-Image Diffusion model, which contains a Low-Resolution Diffusion Model and a Super-Resolution Diffusion Model, can then translate those semantic audio vectors into visual representations of place effectively. We evaluated our proposed model by using both machine-based and human-centered approaches. We proved that the generated street view images align with our common perceptions, and accurately create several key street elements of the original soundscapes. It also demonstrates that soundscapes provide sufficient visual information places. This study stands at the forefront of the intersection between generative AI and human geography, demonstrating how human multi-sensory experiences can be linked. We aim to enrich geospatial data science and AI studies with human experiences. It has the potential to inform multiple domains such as human geography, environmental psychology, and urban design and planning, as well as advancing our knowledge of human-environment relationships.

VisibleSound: Perceiving Environmental Sound with 4D Form

3D Audio Rendering in Distributed Virtual Environment

An Environmental Perception and Navigational Assistance System for Visually Impaired Persons Based on Semantic Stixels and Sound Interaction

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

Perceptual Content Authentication and Tamper Detection of Facial Image

From 3D landscape visualization to environmental simulation: The contribution of sound to the perception of virtual environments

Perceiving Graphical and Pictorial Information via Hearing and Touch

Audiovisual integration system based on synesthesia for people with impaired hearing

Visualization of isomorphism-synesthesia of colour and music

Synesthesia AR: Creating Sound-to-Color Synesthesia in Augmented Reality

Visual to Sound: Generating Natural Sound for Videos in the Wild

Sound Space: Communicating Acoustics Through Interactive Visualization

An Artistic Visualization of Music Modeling a Synesthetic Experience

Three-Dimensional Acoustic Simulation Based on Virtual Environments.

How Does Visual Context of Urban Open Spaces Drive Soundscape Pleasantness and Eventfulness: a Laboratory Experiment Using a Higher Order Ambisonics Speaker System and Immersive Virtual Reality

A Scene Representation for Online Spatial Sonification

From Hearing to Seeing: Linking Auditory and Visual Place Perceptions with Soundscape-to-image Generative Artificial Intelligence

A Perceptual Study of Sound Ecology in Peripheral Sonification

SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning

Help the Blind See: Assistance for the Visually Impaired through Augmented Acoustic Simulation

The Research on Digital Audio-Visual Synesthesia