Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction

Xueyuan Chen,Yuejiao Wang,Xixin Wu,Disong Wang,Zhiyong Wu,Xunying Liu,Helen Meng

2024-01-31

Abstract:Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech by improving the intelligibility and naturalness. This is a challenging task especially for patients with severe dysarthria and speaking in complex, noisy acoustic environments. To address these challenges, we propose a novel multi-modal framework to utilize visual information, e.g., lip movements, in DSR as extra clues for reconstructing the highly abnormal pronunciations. The multi-modal framework consists of: (i) a multi-modal encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual features; (ii) a variance adaptor to infer the normal phoneme duration and pitch contour from the extracted phoneme embeddings; (iii) a speaker encoder to encode the speaker's voice characteristics; and (iv) a mel-decoder to generate the reconstructed mel-spectrogram based on the extracted phoneme embeddings, prosodic features and speaker embeddings. Both objective and subjective evaluations conducted on the commonly used UASpeech corpus show that our proposed approach can achieve significant improvements over baseline systems in terms of speech intelligibility and naturalness, especially for the speakers with more severe symptoms. Compared with original dysarthric speech, the reconstructed speech achieves 42.1\% absolute word error rate reduction for patients with more severe dysarthria levels.

Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is **Dysarthric Speech Reconstruction (DSR)**. Specifically, the goal is to convert the speech of patients with dysarthria into normal speech by improving the clarity and naturalness of the speech. This is a challenging task, especially for those patients with severe dysarthria who speak in complex noisy environments.

To address these challenges, the paper proposes a new multimodal framework that uses visual information (such as lip movements) as an additional cue to reconstruct highly abnormal pronunciations. This multimodal framework mainly consists of the following four parts:

1. **Multimodal Encoder**: Extract robust phoneme embeddings from dysarthric speech and combine auxiliary visual features.
2. **Variance Adapter**: Infer normal phoneme durations and pitch contours from the extracted phoneme embeddings.
3. **Speaker Encoder**: Encode the voice characteristics of the speaker.
4. **Mel Decoder**: Generate a reconstructed Mel - spectrogram based on the extracted phoneme embeddings, prosodic features, and speaker embeddings.

The paper demonstrates significant improvements in the proposed multimodal method in terms of improving speech clarity and naturalness through objective and subjective evaluations on the commonly - used UASpeech corpus, especially for patients with more severe symptoms. Compared with the original dysarthric speech, the reconstructed speech achieves a 42.1% absolute word error rate reduction in patients at the severe dysarthria level.