Glucose production from hydrolysis of cellulose over a novel silica catalyst under hydrothermal conditions.

Huayu Wang,Changbin Zhang,Hong He,Lian Wang

DOI: https://doi.org/10.1016/S1001-0742(11)60795-X

2012-03-01

Journal of Environmental Science

Abstract:

What problem does this paper attempt to address?

VisibleSound: Perceiving Environmental Sound with 4D Form

Fang Tian Ying,Ye Tao,Guan Yun Wang,Qi Wang

DOI: https://doi.org/10.4028/www.scientific.net/amr.718-720.1777

2013-01-01

Advanced Materials Research

Abstract:The progression of application software and high-speed Internet access has, in general, the ability to commonly engage synesthetic experience. However, current synesthesia systems do not meet their full potential, as they are restricted by the simple display of non-intelligent two-dimensional form. This paper is going to present a system for enhancing visual and auditory attributes connection by augmenting the traditional sound visualization system with natural perceptual properties beyond two-dimensional space. A method of expanding the analysis capabilities of the system over changes in the frequencies and sound pressure levels of the live environmental sound stream was explored. With this method, a semantically rich four-dimensional form of sound containing information regarding audio properties and location could be generated. Some specific scenarios are discussed in order to explore visual form of natural perception of environmental sounds, multimedia voice and musical performance.
Exploring Effective Relationships Between Visual-Audio Channels in Data Visualization

Sadia Rubab,Lingyun Yu,Junxiu Tang,Yingcai Wu

DOI: https://doi.org/10.1007/s12650-023-00909-3

IF: 1.7

2023-04-13

Journal of Visualization

Abstract:In recent years, there has been a growing trend towards taking advantage of audio--visual representations. Previous research has aimed at improving users' performance and engagement with these representations. The attainment of these benefits primarily depends on the effectiveness of audio--visual relationships used to represent the data. However, the visualization field yet lacks an empirical study that guides the effective relationships. Given the compatibility effect between visual and auditory channels, this research presents the effectiveness of four audio channels (timbre, pitch, loudness, and tempo) with six visual channels (spatial position, color, position, length, angle, and area). In six experiments, one per visual channel, we observed how each audio channel, when used with a visual channel, impacted users' ability to perform the differentiation or similarity task accurately. Each experiment provided the ranking of audio channels along a visual channel. Central to our experiments was the evaluation at two stages, and accordingly, we identified the effectiveness. Our results showed that timbre, with spatial position and color, aided in more accurate target identification than the three other audio channels. With position and length, pitch allowed a more accurate judgment of the magnitude of data than loudness and tempo but was less accurate than the other two channels along angle and area. Overall, our experiments showed that the choice of representation methods and tasks had impacted the effectiveness of audio channels.

computer science, interdisciplinary applications,imaging science & photographic technology
Sound Space: Communicating Acoustics Through Interactive Visualization

Christopher Morse,Adam Chernick,Zeyu Ren,Sabrina Naumovski,Luke Gehron

DOI: https://doi.org/10.1109/gem.2019.8811535

2019-06-01

Abstract:We present Sound Space, an interactive and engaging environment in which to explore some of the more basic concepts and consequences of acoustic design. By presenting acoustic data visually rather than aurally, we give the user the opportunity to pause, rewind, and fast forward the sound propagation in the space. As an immersive three dimensional visualization, the designer can gain a better spatial understanding of the relationship between a room’s geometry and its acoustic performance. With a focus on both accuracy and design aesthetics, we present acoustic data through a compelling and fun experience in order to increase the user’s engagement and understanding of the data. The resulting visualization is immersive and dynamic, allowing a designer to interact with the space, make changes, and see the consequences in real time. This reduces barriers to acoustic considerations in early design phases, and can aid the sound design of space that would not traditionally have an acoustic focus, but which can still benefit significantly from improved acoustic considerations.
Protein-losing gastroenteropathy. Hypoproteinaemia due to gastrointestinal protein loss of varying aetiology, diagnosed by means of 131-I-albumin.

M. Schwartz,S. Jarnum

1961-03-01

Abstract:
Vision Perceptually Restores Auditory Spectral Dynamics in Speech

John Plass,David Brang,Satoru Suzuki,Marcia Grabowecky

DOI: https://doi.org/10.31234/osf.io/t954p

2019-05-20

Abstract:Visual speech facilitates auditory speech perception, but the visual cues responsible for these effects and the crossmodal information they provide remain unclear. Because visible articulators shape the spectral content of auditory speech, we hypothesized that listeners may be able to extract spectrotemporal information from visual speech to facilitate auditory speech perception. To uncover statistical regularities that could subserve such facilitations, we compared the resonant frequency of the oral cavity to the shape of the oral aperture during speech. We found that the time-frequency dynamics of oral resonances could be recovered with unexpectedly high precision from the shape of the mouth during speech. Because both auditory frequency modulations and visual shape properties are neurally encoded as mid-level perceptual features, we hypothesized that this feature-level correspondence would allow for spectrotemporal information to be recovered from visual speech without reference to higher order (e.g., phonemic) speech representations. Isolating these features from other speech cues, we found that speech-based shape deformations improved sensitivity for corresponding frequency modulations, suggesting that the perceptual system exploits crossmodal correlations in mid-level feature representations to enhance speech perception. To test whether this correspondence could be used to improve comprehension, we selectively degraded the spectral or temporal dimensions of auditory sentence spectrograms to assess how well visual speech facilitated comprehension under each degradation condition. Visual speech produced drastically larger enhancements during spectral degradation, suggesting a condition-specific facilitation effect driven by crossmodal recovery of auditory speech spectra. Visual speech may therefore facilitate perception by crossmodally restoring degraded spectrotemporal signals in speech.
Visualising Model Training via Vowel Space for Text-To-Speech Systems

Binu Abeysinghe,Jesin James,Catherine I. Watson,Felix Marattukalam

DOI: https://doi.org/10.48550/arXiv.2208.09775

2022-08-21

Audio and Speech Processing

Abstract:With the recent developments in speech synthesis via machine learning, this study explores incorporating linguistics knowledge to visualise and evaluate synthetic speech model training. If changes to the first and second formant (in turn, the vowel space) can be seen and heard in synthetic speech, this knowledge can inform speech synthesis technology developers. A speech synthesis model trained on a large General American English database was fine-tuned into a New Zealand English voice to identify if the changes in the vowel space of synthetic speech could be seen and heard. The vowel spaces at different intervals during the fine-tuning were analysed to determine if the model learned the New Zealand English vowel space. Our findings based on vowel space analysis show that we can visualise how a speech synthesis model learns the vowel space of the database it is trained on. Perception tests confirmed that humans could perceive when a speech synthesis model has learned characteristics of the speech database it is training on. Using the vowel space as an intermediary evaluation helps understand what sounds are to be added to the training database and build speech synthesis models based on linguistics knowledge.
Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing

Dhruv Jain,Leah Findlater,Jamie Gilkeson,Benjamin Holland,Ramani Duraiswami,Dmitry Zotkin,Christian Vogler,Jon E. Froehlich

DOI: https://doi.org/10.1145/2702123.2702393

2015-04-18

Abstract:Persons with hearing loss use visual signals such as gestures and lip movement to interpret speech. While hearing aids and cochlear implants can improve sound recognition, they generally do not help the wearer localize sound necessary to leverage these visual cues. In this paper, we design and evaluate visualizations for spatially locating sound on a head-mounted display (HMD). To investigate this design space, we developed eight high-level visual sound feedback dimensions. For each dimension, we created 3-12 example visualizations and evaluated these as a design probe with 24 deaf and hard of hearing participants (Study 1). We then implemented a real-time proof-of-concept HMD prototype and solicited feedback from 4 new participants (Study 2). Study 1 findings reaffirm past work on challenges faced by persons with hearing loss in group conversations, provide support for the general idea of sound awareness visualizations on HMDs, and reveal preferences for specific design options. Although preliminary, Study 2 further contextualizes the design probe and uncovers directions for future work.
Parallel Chords: an audio-visual analytics design for parallel coordinates

Elias Elmquist,Kajetan Enge,Alexander Rind,Carlo Navarra,Robert Höldrich,Michael Iber,Alexander Bock,Anders Ynnerman,Wolfgang Aigner,Niklas Rönnberg

DOI: https://doi.org/10.1007/s00779-024-01795-8

2024-05-09

Personal and Ubiquitous Computing

Abstract:One of the commonly used visualization techniques for multivariate data is the parallel coordinates plot. It provides users with a visual overview of multivariate data and the possibility to interactively explore it. While pattern recognition is a strength of the human visual system, it is also a strength of the auditory system. Inspired by the integration of the visual and auditory perception in everyday life, we introduce an audio-visual analytics design named Parallel Chords combining both visual and auditory displays. Parallel Chords lets users explore multivariate data using both visualization and sonification through the interaction with the axes of a parallel coordinates plot. To illustrate the potential of the design, we present (1) prototypical data patterns where the sonification helps with the identification of correlations, clusters, and outliers, (2) a usage scenario showing the sonification of data from non-adjacent axes, and (3) a controlled experiment on the sensitivity thresholds of participants when distinguishing the strength of correlations. During this controlled experiment, 35 participants used three different display types, the visualization, the sonification, and the combination of these, to identify the strongest out of three correlations. The results show that all three display types enabled the participants to identify the strongest correlation — with visualization resulting in the best sensitivity. The sonification resulted in sensitivities that were independent from the type of displayed correlation, and the combination resulted in increased enjoyability during usage.

computer science, information systems,telecommunications
From 3D landscape visualization to environmental simulation: The contribution of sound to the perception of virtual environments

Mark Lindquist,Eckart Lange,Jian Kang

DOI: https://doi.org/10.1016/j.landurbplan.2015.12.017

IF: 8.119

2016-04-01

Landscape and Urban Planning

Abstract:This research investigated the perceptual interaction of combining sound with 3D landscape visualizations. Images sourced from Google Earth at St. James's Park, London, UK, showing terrain only, terrain with built form or terrain with primarily vegetation were paired with four sound conditions using recordings from the park (i.e. ‘no sound’, anthropogenic, mechanical and natural). Perceived realism and preference were evaluated using a survey delivered via the Internet and in a controlled laboratory environment (N =199 total). Analysis using repeated measures ANOVA indicated the interaction of sound and 3D visualizations significantly alters environmental perception both positively and negatively. Sounds and visuals that are congruent receive higher realism and preference ratings while the more incongruent the combination is, the lower the corresponding ratings. The lowest realism and preference ratings are given to visualizations showing terrain only combined with speech. The highest realism ratings overall correspond to visualization with built form combined with speech, and visualizations showing primarily vegetation paired with a birdcall. The absolute highest realism rating was for the visualization with primarily vegetation and some built form paired with speech, while the highest preference ratings correspond to visualizations showing vegetation paired with birdcall or no sound. Aural-visual data collected via the web-based survey was comparable to data collected in the laboratory and overall realism ratings for the Google Earth visualizations were low (e.g. below 3 on a 1–5 likert type scale). The results suggest there is an opportunity to increase experiential authenticity of 3D landscape visualizations with sound.

environmental studies,ecology,geography, physical,regional & urban planning,urban studies
Visualizing Video Sounds With Sound Word Animation to Enrich User Experience

Fangzhou Wang,Hidehisa Nagano,Kunio Kashino,Takeo Igarashi

DOI: https://doi.org/10.1109/tmm.2016.2613641

IF: 7.3

2017-02-01

IEEE Transactions on Multimedia

Abstract:Sound information in videos plays an important role in shaping the user experience. When sound is not accessible in videos, text captions can provide sound information. However, conventional text captions are not very expressive for nonverbal sounds because they are designed to visualize speech sounds. Here, we present a framework to automatically transform nonverbal video sounds into animated sound words and position them near the sound source objects in the video for visualization. This provides natural visual representation of nonverbal sounds with rich information about the sound category and dynamics. To evaluate how the animated sound words generated by our framework affect the user experience, we implemented an experimental system and conducted a user study involving over 300 participants from an online crowdsourcing service. The results of the user study show that the animated sound words can effectively and naturally visualize the dynamics of sound while clarifying the position of the sound source as well as contribute to making video-watching more enjoyable and increasing the visual impact of videos.

computer science, information systems,telecommunications, software engineering
High‐Pitched Sound is Open and Low‐Pitched Sound is Closed: Representing the Spatial Meaning of Pitch Height

Lari Vainio,Ida‐Lotta Myllylä,Alexandra Wikström,Martti Vainio

DOI: https://doi.org/10.1111/cogs.13486

IF: 2.617

2024-08-20

Cognitive Science

Abstract:Research shows that high‐ and low‐pitch sounds can be associated with various meanings. For example, high‐pitch sounds are associated with small concepts, whereas low‐pitch sounds are associated with large concepts. This study presents three experiments revealing that high‐pitch sounds are also associated with open concepts and opening hand actions, while low‐pitch sounds are associated with closed concepts and closing hand actions. In Experiment 1, this sound‐meaning correspondence effect was shown using the two‐alternative forced‐choice task, while Experiments 2 and 3 used reaction time tasks to show this interaction. In Experiment 2, high‐pitch vocalizations were found to facilitate opening hand gestures, and low‐pitch vocalizations were found to facilitate closing hand gestures, when performed simultaneously. In Experiment 3, high‐pitched vocalizations were produced particularly rapidly when the visual target stimulus presented an open object, and low‐pitched vocalizations were produced particularly rapidly when the target presented a closed object. These findings are discussed concerning the meaning of intonational cues. They are suggested to be based on cross‐modally representing conceptual spatial knowledge in sensory, motor, and affective systems. Additionally, this pitch‐opening effect might share cognitive processes with other pitch‐meaning effects.

psychology, experimental
Humans can use positive and negative spectrotemporal correlations to detect rising and falling pitch

Parisa A Vaziri,Samuel D McDougle,Damon A Clark

DOI: https://doi.org/10.1101/2024.08.03.606481

2024-11-21

Abstract:To discern speech or appreciate music, the human auditory system detects how pitch increases or decreases over time. However, the algorithms used to detect changes in pitch, or pitch motion, are incompletely understood. Here, using psychophysics, computational modeling, functional neuroimaging, and analysis of recorded speech, we ask if humans can detect pitch motion using computations analogous to those used by the visual system. We adapted stimuli from studies of vision to create novel auditory correlated noise stimuli that elicited robust pitch motion percepts. Crucially, these stimuli are inharmonic and possess no persistent features across frequency or time, but do possess positive or negative local spectrotemporal correlations in intensity. In psychophysical experiments, we found clear evidence that humans can judge pitch direction based only on positive or negative spectrotemporal intensity correlations. The key behavioral result -- robust sensitivity to the negative spectrotemporal correlations -- is a direct analogue of illusory "reverse-phi" motion in vision, and thus constitutes a new auditory illusion. Our behavioral results and computational modeling led us to hypothesize that human auditory processing may employ pitch direction opponency. fMRI measurements in auditory cortex supported this hypothesis. To link our psychophysical findings to real-world pitch perception, we analyzed recordings of English and Mandarin speech and found that pitch direction was robustly signaled by both positive and negative spectrotemporal correlations, suggesting that sensitivity to both types of correlations confers ecological benefits. Overall, this work reveals how motion detection algorithms sensitive to local correlations are deployed by the central nervous system across disparate modalities (vision and audition) and dimensions (space and frequency).

Biology
Computer graphics visualization for acoustic simulation

A. Stettner,D. P. Greenberg

DOI: https://doi.org/10.1145/74334.74353

1989-07-01

ACM SIGGRAPH Computer Graphics

Abstract:Computer simulations can be used to generate the spatial and temporal data describing the acoustical behavior of performance halls, but typically the analytical results are difficult to assimilate and compare. By using computer graphics to display the multi-dimensional data, substantially greater amounts of information than that conveyed by standard techniques can be communicated to the designer. This allows designs of different acoustical spaces to be tested, evaluated, and compared.An example comparing the acoustical behavior of three different concert halls demonstrates these techniques and allows for the simultaneous assimilation of much of the information necessary to evaluate the acoustical nature of a space. The use of three-dimensional images, color, animation and abstract representation allows for the comprehension of the complex results of a scientific simulation. Specifically, the simultaneous display of particular icons familiar to the discipline enabled the simultaneous presentation of up to twelve parameters.From a more general point of view, the procedures demonstrate how computer graphics can be utilized for the portrayal of multi-dimensional time dependent data. The visualization techniques are potentially useful for the display of three-dimensional vector fields in many scientific and design applications.
Visualization of Speech Perception Analysis via Phoneme Alignment: A Pilot Study

J. Tilak Ratnanather,Lydia C. Wang,Seung-Ho Bae,Erin R. O'Neill,Elad Sagi,Daniel J. Tward

DOI: https://doi.org/10.3389/fneur.2021.724800

IF: 3.4

2022-01-11

Frontiers in Neurology

Abstract:Objective: Speech tests assess the ability of people with hearing loss to comprehend speech with a hearing aid or cochlear implant. The tests are usually at the word or sentence level. However, few tests analyze errors at the phoneme level. So, there is a need for an automated program to visualize in real time the accuracy of phonemes in these tests. Method: The program reads in stimulus-response pairs and obtains their phonemic representations from an open-source digital pronouncing dictionary. The stimulus phonemes are aligned with the response phonemes via a modification of the Levenshtein Minimum Edit Distance algorithm. Alignment is achieved via dynamic programming with modified costs based on phonological features for insertion, deletions and substitutions. The accuracy for each phoneme is based on the F1-score. Accuracy is visualized with respect to place and manner (consonants) or height (vowels). Confusion matrices for the phonemes are used in an information transfer analysis of ten phonological features. A histogram of the information transfer for the features over a frequency-like range is presented as a phonemegram. Results: The program was applied to two datasets. One consisted of test data at the sentence and word levels. Stimulus-response sentence pairs from six volunteers with different degrees of hearing loss and modes of amplification were analyzed. Four volunteers listened to sentences from a mobile auditory training app while two listened to sentences from a clinical speech test. Stimulus-response word pairs from three lists were also analyzed. The other dataset consisted of published stimulus-response pairs from experiments of 31 participants with cochlear implants listening to 400 Basic English Lexicon sentences via different talkers at four different SNR levels. In all cases, visualization was obtained in real time. Analysis of 12,400 actual and random pairs showed that the program was robust to the nature of the pairs. Conclusion: It is possible to automate the alignment of phonemes extracted from stimulus-response pairs from speech tests in real time. The alignment then makes it possible to visualize the accuracy of responses via phonological features in two ways. Such visualization of phoneme alignment and accuracy could aid clinicians and scientists.

neurosciences,clinical neurology
Low is large: spatial location and pitch interact in voice-based body size estimation

Katarzyna Pisanski,Sari G E Isenstein,Kelyn J Montano,Jillian J M O'Connor,David R Feinberg

DOI: https://doi.org/10.3758/s13414-016-1273-6

Abstract:The binding of incongruent cues poses a challenge for multimodal perception. Indeed, although taller objects emit sounds from higher elevations, low-pitched sounds are perceptually mapped both to large size and to low elevation. In the present study, we examined how these incongruent vertical spatial cues (up is more) and pitch cues (low is large) to size interact, and whether similar biases influence size perception along the horizontal axis. In Experiment 1, we measured listeners' voice-based judgments of human body size using pitch-manipulated voices projected from a high versus a low, and a right versus a left, spatial location. Listeners associated low spatial locations with largeness for lowered-pitch but not for raised-pitch voices, demonstrating that pitch overrode vertical-elevation cues. Listeners associated rightward spatial locations with largeness, regardless of voice pitch. In Experiment 2, listeners performed the task while sitting or standing, allowing us to examine self-referential cues to elevation in size estimation. Listeners associated vertically low and rightward spatial cues with largeness more for lowered- than for raised-pitch voices. These correspondences were robust to sex (of both the voice and the listener) and head elevation (standing or sitting); however, horizontal correspondences were amplified when participants stood. Moreover, when participants were standing, their judgments of how much larger men's voices sounded than women's increased when the voices were projected from the low speaker. Our results provide novel evidence for a multidimensional spatial mapping of pitch that is generalizable to human voices and that affects performance in an indirect, ecologically relevant spatial task (body size estimation). These findings suggest that crossmodal pitch correspondences evoke both low-level and higher-level cognitive processes.
Towards a unified terminology for sonification and visualization

Kajetan Enge,Alexander Rind,Michael Iber,Robert Höldrich,Wolfgang Aigner

DOI: https://doi.org/10.1007/s00779-023-01720-5

2023-08-22

Personal and Ubiquitous Computing

Abstract:Both sonification and visualization convey information about data by effectively using our human perceptual system, but their ways to transform the data differ. Over the past 30 years, the sonification community has demanded a holistic perspective on data representation, including audio-visual analysis, several times. A design theory of audio-visual analysis would be a relevant step in this direction. An indispensable foundation for this endeavor is a terminology describing the combined design space. To build a bridge between the domains, we adopt three of the established theoretical constructs from visualization theory for the field of sonification. The three constructs are the spatial substrate , the visual mark , and the visual channel . In our model, we choose time to be the temporal substrate of sonification. Auditory marks are then positioned in time, such as visual marks are positioned in space. Auditory channels are encoded into auditory marks to convey information. The proposed definitions allow discussing visualization and sonification designs as well as multi-modal designs based on a common terminology. While the identified terminology can support audio-visual analytics research, it also provides a new perspective on sonification theory itself.

computer science, information systems,telecommunications
Real-Time Interactive Visualization Aiding Pronunciation of English as a Second Language

Dorina Dibra,Nuno Otero,Oskar Pettersson

DOI: https://doi.org/10.1109/icalt.2014.131

2014-07-01

Abstract:Some studies have been reporting encouraging results concerning the possibilities of combining voice and its visual representation for language learning. Following this line of investigation this paper explores the potential impact of two distinct visualization styles for the learning of English pronunciation of syllables for non-native speakers: a) highlighting syllables and b) the visualization of the produced sound wave. In order to evaluate the benefits of the two different styles three distinct digital tool prototypes were created in order to test four study conditions. The conditions under evaluation were a) teaching syllables without the support of any digital tool; b) teaching using a prototype that highlighted the syllables under study; c) using a prototype that displayed the sound wave of the syllable to be learnt and d) a prototype that combined the functionality of b) and c). Results suggest that the combined approach seems to be as effective as the traditional classroom approach of teaching the syllables. However, more research is needed in order to consolidate the findings, being able to explore in more detail how is the learning process occurring and to what extend the tools developed can be integrated into classroom practice.
In-situ speech visualization in real-time interactive installation and performance

Golan Levin,Zachary Lieberman

DOI: https://doi.org/10.1145/987657.987659

2004-01-01

Abstract:Although we can sense someone's vocalizations with our ears, nose, and haptic sense, speech is invisible to us without the help of technical aids. In this paper, we present three interactive artworks which explore the question: "if we could see our speech, what might it look like?" The artworks we present are concerned with the aesthetic implications of making the human voice visible, and were created with a particular emphasis on interaction designs that support the perception of tight spatio-temporal relationships between sound, image, and the body. We coin the term in-situ speech visualization to describe a variety of augmented-reality techniques by which graphic representations of speech can be made to appear coincident with their apparent point of origination.
Climate data sonification and visualization: An analysis of topics, aesthetics, and characteristics in 32 recent projects

PerMagnus Lindborg,Sara Lenzi,Manni Chen

DOI: https://doi.org/10.3389/fpsyg.2022.1020102

IF: 3.8

2023-01-26

Frontiers in Psychology

Abstract:Introduction: It has proven a hard challenge to stimulate climate action with climate data. While scientists communicate through words, numbers, and diagrams, artists use movement, images, and sound. Sonification, the translation of data into sound, and visualization, offer techniques for representing climate data with often innovative and exciting results. The concept of sonification was initially defined in terms of engineering, and while this view remains dominant, researchers increasingly make use of knowledge from electroacoustic music (EAM) to make sonifications more convincing. Methods: The Aesthetic Perspective Space (APS) is a two-dimensional model that bridges utilitarian-oriented sonification and music. We started with a review of 395 sonification projects, from which a corpus of 32 that target climate change was chosen; a subset of 18 also integrate visualization of the data. To clarify relationships with climate data sources, we determined topics and subtopics in a hierarchical classification. Media duration and lexical diversity in descriptions were determined. We developed a protocol to span the APS dimensions, Intentionality and Indexicality, and evaluated its circumplexity. Results: We constructed 25 scales to cover a range of qualitative characteristics applicable to sonification and sonification-visualization projects, and through exploratory factor analysis, identified five essential aspects of the project descriptions, labeled Action, Technical, Context, Perspective, and Visualization. Through linear regression modeling, we investigated the prediction of aesthetic perspective from essential aspects, media duration, and lexical diversity. Significant regressions across the corpus were identified for Perspective (ß = 0.41 *** ) and lexical diversity (ß = −0.23 * ) on Intentionality, and for Perspective (ß = 0.36 *** ) and Duration (logarithmic; ß = −0.25 * ) on Indexicality. Discussion: We discuss how these relationships play out in specific projects, also within the corpus subset that integrated data visualization, as well as broader implications of aesthetics on design techniques for multimodal representations aimed at conveying scientific data. Our approach is informed by the ongoing discussion in sound design and auditory perception research communities on the relationship between sonification and EAM. Through its analysis of topics, qualitative characteristics, and aesthetics across a range of projects, our study contributes to the development of empirically founded design techniques, applicable to climate science communication and other fields.

psychology, multidisciplinary
Visualization of isomorphism-synesthesia of colour and music

Xue Mao,Jie Xu,Jiahong Lang,Shangshu Zhang

DOI: https://doi.org/10.1016/j.visinf.2023.10.006

2023-11-05

Visual Informatics

Abstract:Music and colour, as human hearing and visual art, are closely related to human psychological feelings and symbolic associations. There is an isomorphic relationship between music and colour. The article uses the concept of "synesthesia" in psychology and the "co-construction" relationship in mathematics as a bridge, based on Kandinsky's "inner sound" theory and Mallion's "tone-colour system", an interdisciplinary theoretical model of "timbre isomorphism synesthesia" (ISCM) is constructed. At the practical level, based on the ISCM theory, a set of timbre synesthesia visualization tool ASAH and visualization process are designed, through music data input, graphics mapping visualization, colour mapping visualization, real-time interactive visualization, and finally output timbre synesthesia visualization works. In order to avoid the visual homogenization caused by algorithm design, ASHA has set up a custom editor, which emphasizes the individual differences and multi-sensory experience of tonal synesthesia visualization.

computer science, information systems, interdisciplinary applications, software engineering

Glucose production from hydrolysis of cellulose over a novel silica catalyst under hydrothermal conditions.

VisibleSound: Perceiving Environmental Sound with 4D Form

Exploring Effective Relationships Between Visual-Audio Channels in Data Visualization

Sound Space: Communicating Acoustics Through Interactive Visualization

Protein-losing gastroenteropathy. Hypoproteinaemia due to gastrointestinal protein loss of varying aetiology, diagnosed by means of 131-I-albumin.

Vision Perceptually Restores Auditory Spectral Dynamics in Speech

Visualising Model Training via Vowel Space for Text-To-Speech Systems

Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing

Parallel Chords: an audio-visual analytics design for parallel coordinates

From 3D landscape visualization to environmental simulation: The contribution of sound to the perception of virtual environments

Visualizing Video Sounds With Sound Word Animation to Enrich User Experience

High‐Pitched Sound is Open and Low‐Pitched Sound is Closed: Representing the Spatial Meaning of Pitch Height

Humans can use positive and negative spectrotemporal correlations to detect rising and falling pitch

Computer graphics visualization for acoustic simulation

Visualization of Speech Perception Analysis via Phoneme Alignment: A Pilot Study

Low is large: spatial location and pitch interact in voice-based body size estimation

Towards a unified terminology for sonification and visualization

Real-Time Interactive Visualization Aiding Pronunciation of English as a Second Language

In-situ speech visualization in real-time interactive installation and performance

Climate data sonification and visualization: An analysis of topics, aesthetics, and characteristics in 32 recent projects

Visualization of isomorphism-synesthesia of colour and music