Abstract:In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, the impartial and replicable evaluation of these ASR systems encounters challenges due to various crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a general-purpose, open-source platform designed for ASR evaluation. With this platform: (i) We report a comprehensive benchmark, unveiling the current state-of-the-art panorama for ASR systems, covering both open-source models and industrial commercial services. (ii) We quantize how distinct nuances in the scoring pipeline influence the final benchmark outcomes. These include nuances related to capitalization, punctuation, interjection, contraction, synonym usage, compound words, etc. These issues have gained prominence in the context of the transition towards an End-to-End future. (iii) We propose a practical modification to the conventional Token-Error-Rate (TER) evaluation metric, with inspirations from Kolmogorov complexity and Normalized Information Distance (NID). This adaptation, called modified-TER (mTER), achieves proper normalization and symmetrical treatment of reference and hypothesis. By leveraging this platform as a large-scale testing ground, this study demonstrates the robustness and backward compatibility of mTER when compared to TER. The SpeechColab Leaderboard is accessible at

SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation

Interpretable Multi-dataset Evaluation for Named Entity Recognition

An End-to-End System for Reproducibility Assessment of Source Code Repositories via Their Readmes

Debiasing Generative Named Entity Recognition by Calibrating Sequence Likelihood.

Toward Human-Like Evaluation for Natural Language Generation with Error Analysis

Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP

CEScore: Simple and Efficient Confidence Estimation Model for Evaluating Split and Rephrase

Generalisation in Named Entity Recognition: A Quantitative Analysis

mlscorecheck : Testing the consistency of reported performance scores and experiments in machine learning

Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis

SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation

Neural Reranking for Named Entity Recognition

Statistical Dataset Evaluation: A Case Study on Named Entity Recognition

Reproducibility in NLP: What Have We Learned from the Checklist?

Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks

INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback

Finding Replicable Human Evaluations via Stable Ranking Probability

Iterative Strategy for Named Entity Recognition with Imperfect Annotations.

RNA-SeQC: RNA-seq metrics for quality control and process optimization

S-NER: A Concise and Efficient Span-Based Model for Named Entity Recognition

Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores