Abstract:Recently, Conformer as a backbone network for end-to-end automatic speech recognition achieved state-of-the-art performance. The Conformer block leverages a self-attention mechanism to capture global information, along with a convolutional neural network to capture local information, resulting in improved performance. However, the Conformer-based model encounters an issue with the self-attention mechanism, as computational complexity grows quadratically with the length of the input sequence. Inspired by previous Connectionist Temporal Classification (CTC) guided blank skipping during decoding, we introduce intermediate CTC outputs as guidance into the downsampling procedure of the Conformer encoder. We define the frame with non-blank output as key frame. Specifically, we introduce the key frame-based self-attention (KFSA) mechanism, a novel method to reduce the computation of the self-attention mechanism using key frames. The structure of our proposed approach comprises two encoders. Following the initial encoder, we introduce an intermediate CTC loss function to compute the label frame, enabling us to extract the key frames and blank frames for KFSA. Furthermore, we introduce the key frame-based downsampling (KFDS) mechanism to operate on high-dimensional acoustic features directly and drop the frames corresponding to blank labels, which results in new acoustic feature sequences as input to the second encoder. By using the proposed method, which achieves comparable or higher performance than vanilla Conformer and other similar work such as Efficient Conformer. Meantime, our proposed method can discard more than 60\% useless frames during model training and inference, which will accelerate the inference speed significantly. This work code is available in {<a class="link-external link-https" href="https://github.com/scufan1990/Key-Frame-Mechanism-For-Efficient-Conformer" rel="external noopener nofollow">this https URL</a>}

Research on an improved Conformer end-to-end Speech Recognition Model with R-Drop Structure

Nextformer: A ConvNeXt Augmented Conformer For End-To-End Speech Recognition

Conformer-1: Robust ASR via Large-Scale Semisupervised Bootstrapping

An Improvement to Conformer-Based Model for High-Accuracy Speech Feature Extraction and Learning

Efficient End-to-End Speech Recognition Using Performers in Conformers

Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognition

Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition

Continual Learning for On-Device Speech Recognition using Disentangled Conformers

FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition

Self-consistent context aware conformer transducer for speech recognition

CONTINUOUS SPEECH SEPARATION WITH CONFORMER

Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers

Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognition

Conformer Parrotron: a Faster and Stronger End-to-end Speech Conversion and Recognition Model for Atypical Speech

A comparative analysis between Conformer-Transducer, Whisper, and wav2vec2 for improving the child speech recognition

Multi-Convformer: Extending Conformer with Multiple Convolution Kernels

Multi-pass Training and Cross-information Fusion for Low-resource End-to-end Accented Speech Recognition

Efficient conformer-based speech recognition with linear attention

Conformer-based End-to-end Speech Recognition With Rotary Position Embedding

Research on Named Entity Recognition in Improved transformer with R-Drop structure

Exploring RNN-Transducer for Chinese Speech Recognition