Abstract:This thesis focuses on representation learning for sequence data over time or space, aiming to improve downstream sequence prediction tasks by using the learned representations. Supervised learning has been the most dominant approach for training deep neural networks for learning good sequential representations. However, one limiting factor to scale supervised learning is the lack of enough annotated data. Motivated by this challenge, it is natural to explore representation learning methods that can utilize large amounts of unlabeled and weakly labeled data, as well as an additional data modality. I describe my broad study of representation learning for speech data. Unlike most other works that focus on a single learning setting, this thesis studies multiple settings: supervised learning with auxiliary losses, unsupervised learning, semi-supervised learning, and multi-view learning. Besides different learning problems, I also explore multiple approaches for representation learning. Though I focus on speech data, the methods described in this thesis can also be applied to other domains. Overall, the field of representation learning is developing rapidly. State-of-the-art results on speech related tasks are typically based on Transformers pre-trained with large-scale self-supervised learning, which aims to learn generic representations that can benefit multiple downstream tasks. Since 2020, large-scale pre-training has been the de facto choice to achieve good performance. This delayed thesis does not attempt to summarize and compare with the latest results on speech representation learning; instead, it presents a unique study on speech representation learning before the Transformer era, that covers multiple learning settings. Some of the findings in this thesis can still be useful today.

Self-Supervised Speech Representation Learning: A Review

Self-Supervised Representation Learning: Introduction, advances, and challenges

Self-supervised Representation Learning for Speech Processing

Efficiency-oriented approaches for self-supervised speech representation learning

A Survey on Self-Supervised Representation Learning

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends

Self-supervised Reflective Learning through Self-distillation and Online Clustering for Speaker Representation Learning

Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective

Overview of Speaker Modeling and Its Applications: From the Lens of Deep Speaker Representation Learning

Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision

Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks

Speech representation learning: Learning bidirectional encoders with single-view, multi-view, and multi-task methods

Editorial Editorial of Special Issue on Self-Supervised Learning for Speech and Audio Processing

Progressive Multi-scale Self-supervised Learning for Speech Recognition

A Comprehensive Study on Self-Supervised Distillation for Speaker Representation Learning

Sustainable self-supervised learning for speech representations

Understanding Self-Supervised Learning of Speech Representation via Invariance and Redundancy Reduction

Self-supervised Speech Representations Still Struggle with African American Vernacular English

Large-Scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification