Abstract:End-to-end (E2E) spoken language understanding (SLU) is constrained by the cost of collecting speech-semantics pairs, especially when label domains change. Hence, we explore \textit{zero-shot} E2E SLU, which learns E2E SLU without speech-semantics pairs, instead using only speech-text and text-semantics pairs. Previous work achieved zero-shot by pseudolabeling all speech-text transcripts with a natural language understanding (NLU) model learned on text-semantics corpora. However, this method requires the domains of speech-text and text-semantics to match, which often mismatch due to separate collections. Furthermore, using the entire collected speech-text corpus from any domains leads to \textit{imbalance} and \textit{noise} issues. To address these, we propose \textit{cross-modal selective self-training} (CMSST). CMSST tackles imbalance by clustering in a joint space of the three modalities (speech, text, and semantics) and handles label noise with a selection network. We also introduce two benchmarks for zero-shot E2E SLU, covering matched and found speech (mismatched) settings. Experiments show that CMSST improves performance in both two settings, with significantly reduced sample sizes and training time. Our code and data are released in <a class="link-external link-https" href="https://github.com/amazon-science/zero-shot-E2E-slu" rel="external noopener nofollow">this https URL</a>.

Multi-label Masked Language Modeling on Zero-shot Code-switched Sentiment Analysis

Dual Collaborative Visual-Semantic Mapping for Multi-Label Zero-Shot Image Recognition

A Masked Segmental Language Model for Unsupervised Natural Language Segmentation

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

The Model Arena for Cross-lingual Sentiment Analysis: A Comparative Study in the Era of Large Language Models

Multi-modal Generative Adversarial Network for Zero-Shot Learning

Towards Zero-Shot Code-Switched Speech Recognition

HC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding

Progressive Sentiment Analysis for Code-Switched Text Data

Universal Sentence Representation Learning with Conditional Masked Language Model

Zero-shot Cross-lingual Conversational Semantic Role Labeling

Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon

Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization

Adapting deep learning for sentiment classification of code-switched informal short text

Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training

Evaluating and explaining training strategies for zero-shot cross-lingual news sentiment analysis

Simple yet Effective Code-Switching Language Identification with Multitask Pre-Training and Transfer Learning

Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switching

Transductive Multi-label Zero-shot Learning.

Code-mixed Sentiment and Hate-speech Prediction

Unsupervised Self-Training for Sentiment Analysis of Code-Switched Data