Abstract:Domain-specific data is the crux of the successful transfer of machine learning systems from benchmarks to real life. In simple problems such as image classification, crowdsourcing has become one of the standard tools for cheap and time-efficient data collection: thanks in large part to advances in research on aggregation methods. However, the applicability of crowdsourcing to more complex tasks (e.g., speech recognition) remains limited due to the lack of principled aggregation methods for these modalities. The main obstacle towards designing aggregation methods for more advanced applications is the absence of training data, and in this work, we focus on bridging this gap in speech recognition. For this, we collect and release CrowdSpeech -- the first publicly available large-scale dataset of crowdsourced audio transcriptions. Evaluation of existing and novel aggregation methods on our data shows room for improvement, suggesting that our work may entail the design of better algorithms. At a higher level, we also contribute to the more general challenge of developing the methodology for reliable data collection via crowdsourcing. In that, we design a principled pipeline for constructing datasets of crowdsourced audio transcriptions in any novel domain. We show its applicability on an under-resourced language by constructing VoxDIY -- a counterpart of CrowdSpeech for the Russian language. We also release the code that allows a full replication of our data collection pipeline and share various insights on best practices of data collection via crowdsourcing.

Assembling Chinese-Mongolian Speech Corpus via Crowdsourcing.

Utilizing Crowdsourcing for the Construction of Chinese-Mongolian Speech Corpus with Evaluation Mechanism

IMUT-MC: a Speech Corpus for Mongolian Speech Recognition

CNAMD Corpus: A Chinese Natural Audiovisual Multimodal Database of Conversations for Social Interactive Agents

Crowd-Answering System Via Microblogging

Extending the MPC Corpus to Chinese and Urdu - A Multiparty Multi-Lingual Chat Corpus for Modeling Social Phenomena in Language.

Building Uyghur Language Event Annotation Corpus with Crowdsourcing

Constructing a Chinese Conversation Corpus for Sentiment Analysis.

Automatic Construction of Parallel Dialogue Corpora with Rich Information

Multimedia Simultaneous Translation System for Minority Language Communication with Mandarin

TeleCrowd: A Crowdsourcing Approach to Create Informal to Formal Text Corpora

MC$^2$: Towards Transparent and Culturally-Aware NLP for Minority Languages in China

Construction of A Mongolian Telephone Speech Corpus

A Japanese-Chinese Parallel Corpus Using Crowdsourcing for Web Mining

Building a Non-native Speech Corpus Featuring Chinese-English Bilingual Children: Compilation and Rationale

Comparative Study for Multi-Speaker Mongolian TTS with a New Corpus

Building Large Chinese Corpus for Spoken Dialogue Research in Specific Domains.

A Large-Scale Depth-Based Multimodal Audio-Visual Corpus in Mandarin

CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription

Milimili. Collecting Parallel Data via Crowdsourcing

Analysis and Research Based on the Crowdsourcing Corpus System in Guangdong-Hong Kong-Macao Greater Bay Area (GBA)