Abstract:At present, most multi-dialect speech recognition models are based on a hard-parameter-sharing multi-task structure, which makes it difficult to reveal how one task contributes to others. In addition, in order to balance multi-task learning, the weights of the multi-task objective function need to be manually adjusted. This makes multi-task learning very difficult and costly because it requires constantly trying various combinations of weights to determine the optimal task weights. In this paper, we propose a multi-dialect acoustic model that combines soft-parameter-sharing multi-task learning with Transformer, and introduce several auxiliary cross-attentions to enable the auxiliary task (dialect ID recognition) to provide dialect information for the multi-dialect speech recognition task. Furthermore, we use the adaptive cross-entropy loss function as the multi-task objective function, which automatically balances the learning of the multi-task model according to the loss proportion of each task during the training process. Therefore, the optimal weight combination can be found without any manual intervention. Finally, for the two tasks of multi-dialect (including low-resource dialect) speech recognition and dialect ID recognition, the experimental results show that, compared with single-dialect Transformer, single-task multi-dialect Transformer, and multi-task Transformer with hard parameter sharing, our method significantly reduces the average syllable error rate of Tibetan multi-dialect speech recognition and the character error rate of Chinese multi-dialect speech recognition.

Focal Loss for End-to-end Short Utterances Chinese Dialect Identification.

Focal CTC Loss for Chinese Optical Character Recognition on Unbalanced Datasets

Self-attention Based Speaker Recognition Using Cluster-Range Loss

Weighted Cluster-Range Loss and Criticality-Enhancement Loss for Speaker Recognition

Focal Loss for Punctuation Prediction.

GE2E-AC: Generalized End-to-End Loss Training for Accent Classification

Chinese Dialect Speech Recognition Based on End-to-end Machine Learning

Reliable accent specific unit generation with dynamic Gaussian mixture selection for multi-accent speech recognition

Lattice Based Transcription Loss for End-to-End Speech Recognition

Deep Discriminative Feature Learning for Accent Recognition

Local Mismatch Phone for Confidence Measure in Standard and Accented Chinese Speech Recognition

Learning Discriminative Features from Spectrograms Using Center Loss for Speech Emotion Recognition.

A Language Prior Based Focal Loss for Visual Question Answering

Improved Accent Classification Combining Phonetic Vowels with Acoustic Features

Multi-Task Transformer with Adaptive Cross-Entropy Loss for Multi-Dialect Speech Recognition

Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition

Improving Accent Conversion with Reference Encoder and End-To-End Text-To-Speech

A Spectral-change-aware Loss Function for DNN-based Speech Separation.

Zero-Shot Accent Conversion using Pseudo Siamese Disentanglement Network

Mismatched Feature Detection with Finer Granularity for Emotional Speaker Recognition.

Focal Kl-Divergence Based Dilated Convolutional Neural Networks for Co-Channel Speaker Identification.