Speech-based Mark for Data Sonification

Yichun Zhao,Jingyi Lu,Miguel A Nacenta

DOI: https://doi.org/10.1145/3663548.3688514

2024-08-13

Abstract:Sonification serves as a powerful tool for data accessibility, especially for people with vision loss. Among various modalities, speech is a familiar means of communication similar to the role of text in visualization. However, speech-based sonification is underexplored. We introduce SpeechTone, a novel speech-based mark for data sonification and extension to the existing Erie declarative grammar for sonification. It encodes data into speech attributes such as pitch, speed, voice and speech content. We demonstrate the efficacy of SpeechTone through three examples.

Human-Computer Interaction

What problem does this paper attempt to address?

The paper primarily aims to address the issue of data accessibility, particularly the challenge of data access for visually impaired users. The authors propose a speech-based data sonification markup, called SpeechTone, as an extension of the Erie declarative grammar, with the goal of enhancing the expressive capability of data sonification.

Specifically, SpeechTone achieves data sonification by encoding data into speech attributes such as pitch, speed, timbre, and text content. This method leverages speech as a familiar mode of daily communication, allowing visually impaired individuals to interact with data more intuitively. Additionally, by extending the Erie declarative grammar, SpeechTone simplifies the data sonification process and enhances accessibility for users who are not familiar with sound synthesis methods.

To demonstrate the effectiveness of SpeechTone, the paper provides three example demonstrations, each showcasing how different speech attributes (e.g., pitch, speech rate) can be adjusted to convey different types of data information (e.g., the number of car models, trends over years, fuel efficiency). These examples illustrate the potential of SpeechTone in various data analysis tasks.

In summary, the goal of this research is to improve the accessibility and comprehensibility of data for visually impaired groups by developing a new speech sonification technology, namely SpeechTone.