Abstract:A cornerstone in AI research has been the creation and adoption of standardized training and test datasets to earmark the progress of state-of-the-art models. A particularly successful example is the GLUE dataset for training and evaluating Natural Language Understanding (NLU) models for English. The large body of research around self-supervised BERT-based language models revolved around performance improvements on NLU tasks in GLUE. To evaluate language models in other languages, several language-specific GLUE datasets were created. The area of speech language understanding (SLU) has followed a similar trajectory. The success of large self-supervised models such as wav2vec2 enable creation of speech models with relatively easy to access unlabelled data. These models can then be evaluated on SLU tasks, such as the SUPERB benchmark. In this work, we extend this to Indic languages by releasing the IndicSUPERB benchmark. Specifically, we make the following three contributions. (i) We collect Kathbath containing 1,684 hours of labelled speech data across 12 Indian languages from 1,218 contributors located in 203 districts in India. (ii) Using Kathbath, we create benchmarks across 6 speech tasks: Automatic Speech Recognition, Speaker Verification, Speaker Identification (mono/multi), Language Identification, Query By Example, and Keyword Spotting for 12 languages. (iii) On the released benchmarks, we train and evaluate different self-supervised models alongside a commonly used baseline FBANK. We show that language-specific fine-tuned models are more accurate than baseline on most of the tasks, including a large gap of 76\% for the Language Identification task. However, for speaker identification, self-supervised models trained on large datasets demonstrate an advantage. We hope IndicSUPERB contributes to the progress of developing speech language understanding models for Indian languages.

IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation

IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

IndoNLI: A Natural Language Inference Dataset for Indonesian

NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages

SNLI Indo: A recognizing textual entailment dataset in Indonesian derived from the Stanford Natural Language Inference dataset

IndoToD: A Multi-Domain Indonesian Benchmark For End-to-End Task-Oriented Dialogue Systems

NusaBERT: Teaching IndoBERT to be Multilingual and Multicultural

One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia

IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languages

NusaCrowd: A Call for Open and Reproducible NLP Research in Indonesian Languages

Replicable Benchmarking of Neural Machine Translation (NMT) on Low-Resource Local Languages in Indonesia

Constructing and Expanding Low-Resource and Underrepresented Parallel Datasets for Indonesian Local Languages

IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages

Building Dialogue Understanding Models for Low-resource Language Indonesian from Scratch

MILU: A Multi-task Indic Language Understanding Benchmark

IndoSum: A New Benchmark Dataset for Indonesian Text Summarization

Hybrid Models for Emotion Classification and Sentiment Analysis in Indonesian Language

IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian languages

Domain-Specific Language Model Post-Training for Indonesian Financial NLP

DriveThru: a Document Extraction Platform and Benchmark Datasets for Indonesian Local Language Archives