Abstract:Languages have long been described according to their perceived rhythmic attributes. The associated typologies are of interest in psycholinguistics as they partly predict newborns' abilities to discriminate between languages and provide insights into how adult listeners process non-native languages. Despite the relative success of rhythm metrics in supporting the existence of linguistic rhythmic classes, quantitative studies have yet to capture the full complexity of temporal regularities associated with speech rhythm. We argue that deep learning offers a powerful pattern-recognition approach to advance the characterization of the acoustic bases of speech rhythm. To explore this hypothesis, we trained a medium-sized recurrent neural network on a language identification task over a large database of speech recordings in 21 languages. The network had access to the amplitude envelopes and a variable identifying the voiced segments, assuming that this signal would poorly convey phonetic information but preserve prosodic features. The network was able to identify the language of 10-second recordings in 40% of the cases, and the language was in the top-3 guesses in two-thirds of the cases. Visualization methods show that representations built from the network activations are consistent with speech rhythm typologies, although the resulting maps are more complex than two separated clusters between stress and syllable-timed languages. We further analyzed the model by identifying correlations between network activations and known speech rhythm metrics. The findings illustrate the potential of deep learning tools to advance our understanding of speech rhythm through the identification and exploration of linguistically relevant acoustic feature spaces.

Generating EEG features from Acoustic features

Predicting Different Acoustic Features from EEG and towards direct synthesis of Audio Waveform from EEG

Improving EEG based continuous speech recognition using GAN

Speech Synthesis using EEG

Spoken Speech Enhancement using EEG

Generative Adversarial Network (GAN) for Simulating Electroencephalography

Predicting Video features from EEG and Vice versa

Acoustic characterization of speech rhythm: going beyond metrics with recurrent neural networks

Generative adversarial networks in EEG analysis: an overview

Generative Listener EEG for Speech Emotion Recognition Using Generative Adversarial Networks with Compressed Sensing

Electroencephalographic Signal Data Augmentation Based on Improved Generative Adversarial Network

Recurrent Neural Networks and Acoustic Features for Frame-Level Signal-to-Noise Ratio Estimation.

Articulatory-to-Acoustic Conversion with Cascaded Prediction of Spectral and Excitation Features Using Neural Networks

Enhanced Generative Adversarial Networks for Unseen Word Generation from EEG Signals

A Visual Characteristic Land-Scape Design for EEG Signal Based on LSTM-GAN

Towards EEG Generation Using GANs for BCI Applications

Emotion Recognition Based on EEG Using Generative Adversarial Nets and Convolutional Neural Network

Joint Acoustic Factor Learning for Robust Deep Neural Network Based Automatic Speech Recognition

Non-linear prediction with LSTM recurrent neural networks for acoustic novelty detection

Speaker Identification using EEG

Eeg2vec: Self-Supervised Electroencephalographic Representation Learning