Abstract:Transformers have evolved with great success in various artificial intelligence tasks. Thanks to our recent prevalence of self-attention mechanisms, which capture long-term dependency, phenomenal outcomes in speech processing and recognition tasks have been produced. The paper presents a comprehensive survey of transformer techniques oriented in speech modality. The main contents of this survey include (1) background of traditional ASR, end-to-end transformer ecosystem, and speech transformers (2) foundational models in a speech via lingualism paradigm, i.e., monolingual, bilingual, multilingual, and cross-lingual (3) dataset and languages, acoustic features, architecture, decoding, and evaluation metric from a specific topological lingualism perspective (4) popular speech transformer toolkit for building end-to-end ASR systems. Finally, highlight the discussion of open challenges and potential research directions for the community to conduct further research in this domain.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges faced by Automatic Speech Recognition (ASR) systems when dealing with multilingual environments, especially their applications in low - resource languages and cross - language scenarios. Specifically, the objectives of the paper include: 1. **Limitations of traditional ASR systems**: Traditional ASR systems rely on manually - designed features to extract language content from speech signals, lacking in - depth understanding of languages and being difficult to adapt to language diversity. These systems are usually based on Hidden Markov Models (HMM) and Gaussian Mixture Models (GMM), and have deficiencies in computational efficiency, scalability and adaptability when dealing with large - scale datasets, new speech patterns, accents or noisy environments. 2. **Advantages of end - to - end neural network architectures**: In order to overcome the above problems, end - to - end neural network architectures (such as Transformer) have been widely used in recent years. These models can directly map from speech to text and can efficiently transfer knowledge and learning to different languages and dialects. In particular, the Transformer architecture can capture long - term dependencies by introducing the self - attention mechanism, thus achieving remarkable results in speech processing tasks. 3. **Improvement of multilingual and cross - language capabilities**: The paper focuses on how to build ASR systems with multilingual and cross - language capabilities to deal with the differences and complexity between different languages. Specifically, the paper explores the current situation and development directions of monolingual, bilingual, multilingual and cross - language ASR systems, aiming to improve the accuracy and robustness of these systems when dealing with multiple languages. 4. **Support for low - resource languages**: For resource - scarce languages (such as endangered languages), the paper emphasizes the importance of transferring knowledge from resource - rich languages to low - resource languages through techniques such as transfer learning, in order to promote the protection and development of these languages. ### Specific problem summary - **How to improve the multilingual processing ability of ASR systems**: Especially when dealing with multiple languages and their variants, how to ensure the accuracy and robustness of the system. - **How to deal with the challenges of low - resource languages**: For languages lacking a large amount of training data, how to use techniques such as transfer learning for effective ASR modeling. - **How to optimize the Transformer architecture to better process speech signals**: Including how to improve the self - attention mechanism and position encoding to capture long - term dependencies and context information in speech signals. By solving these problems, the paper aims to promote the development of ASR technology, enabling it to be more widely applied in multilingual environments and providing better support for low - resource languages.

Speech Recognition Transformers: Topological-lingualism Perspective

Transformers in Speech Processing: A Survey

Research Status and Prospect of Transformer in Speech Recognition

A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks

Transformers in computational visual media: A survey

Multiresolution and Multimodal Speech Recognition with Transformers

Transformers in Natural Language Processing: A Comprehensive Review

Vision Language Transformers: A Survey

Introduction to Transformers: an NLP Perspective

Survey: Transformer-based Models in Data Modality Conversion

Transformers with convolutional context for ASR

Transformers in Vision: A Survey

A Bidirectional Context Embedding Transformer for Automatic Speech Recognition

Dawn of the transformer era in speech emotion recognition: closing the valence gap

Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision

From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation

What Formal Languages Can Transformers Express? A Survey

Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition

A Survey on Transformers in Reinforcement Learning

Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey

Advancements in Natural language Processing: An In-depth Review of Language Transformer Models