Abstract:Wav2vec 2.0 is an end-to-end framework of self-supervised learning for speech representation that is successful in automatic speech recognition (ASR), but most of the work on the topic has been developed with a single language: English. Therefore, it is unclear whether the self-supervised framework is effective in recognizing other languages with different writing systems, such as Korean which uses the Hangul having a unique writing system. In this paper, we present K-Wav2Vec 2.0, which is a modified version of Wav2vec 2.0 designed for Korean automatic speech recognition by exploring and optimizing various factors of the original Wav2vec 2.0. In fine-tuning, we propose a multi-task hierarchical architecture to reflect the Korean writing structure. Moreover, a joint decoder is applied to alleviate the problem of words existing outside of the vocabulary. In pre-training, we attempted the cross-lingual transfer of the pre-trained model by further pre-training the English Wav2vec 2.0 on a Korean dataset, considering limited resources. Our experimental results demonstrate that the proposed method yields the best performance on both Korean ASR datasets: Ksponspeech (a large-scale Korean speech corpus) and Clovacall (a call-based dialog corpus). Further pre-training is also effective in language adaptation, leading to large improvements without additional data.

Phonological modeling for continuous speech recognition in Korean

Phoneme-level speech and natural language intergration for agglutinative languages

A usage of the syllable unit based on morphological statistics in Korean large vocabulary continuous speech recognition system

Improving Korean NLP Tasks with Linguistically Informed Subword Tokenization and Sub-character Decomposition

Leveraging the Interplay Between Syntactic and Acoustic Cues for Optimizing Korean TTS Pause Formation

Incorporating L2 Phonemes Using Articulatory Features for Robust Speech Recognition

Speech Feature Mapping based on Switching Linear Dynamic System

The Hidden Markov Model of co-articulation and its application to the continuous speech recognition

Context Dependent Syllable Acoustic Model For Continuous Chinese Speech Recognition

A New Acoustic Modeling of Inter-Syllable Context-Dependent Units for Putonghua Continuous Speech Recognition

High Fidelity Text-to-Speech Via Discrete Tokens Using Token Transducer and Group Masked Language Model

TAKTAG: Two-phase learning method for hybrid statistical/rule-based part-of-speech disambiguation

Automatic Speech Recognition (ASR) for the Diagnosis of pronunciation of Speech Sound Disorders in Korean children

Continuous Speech Tokenizer in Text To Speech

Inappropriate Pause Detection In Dysarthric Speech Using Large-Scale Speech Recognition

Research on Context-Dependent Acoustical Unit (Triphone) for Mandarin Continuous Speech Recognition

K-Wav2vec 2.0: Automatic Speech Recognition based on Joint Decoding of Graphemes and Syllables

Adversarially Trained End-to-end Korean Singing Voice Synthesis System

Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis

Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation

Comparison of L2 Korean pronunciation error patterns from five L1 backgrounds by using automatic phonetic transcription