Abstract:We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets generated by slowly-evolving momentum encoders. Driven by the inherent differences between video and audio, our design is asymmetric w.r.t. the two modalities' pretext tasks: Whereas the auditory stream predicts both the visual and auditory targets, the visual one predicts only the auditory targets. We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained. Notably, RAVEn surpasses all self-supervised methods on visual speech recognition (VSR) on LRS3, and combining RAVEn with self-training using only 30 hours of labelled data even outperforms a recent semi-supervised method trained on 90,000 hours of non-public data. At the same time, we achieve state-of-the-art results in the LRS3 low-resource setting for auditory speech recognition (as well as for VSR). Our findings point to the viability of learning powerful speech representations entirely from raw video and audio, i.e., without relying on handcrafted features. Code and models are available at <a class="link-external link-https" href="https://github.com/ahaliassos/raven" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to jointly learn visual and auditory speech representations from original video and audio data to improve the performance of visual speech recognition (VSR) and auditory speech recognition (ASR) in low - resource and high - resource labeled data settings?** ### Specific problem description: 1. **Limitations of existing methods**: - Most of the existing self - supervised methods can only handle word - level samples rather than continuous speech. - Some methods rely on hand - designed features (such as spectrograms or MFCCs), which may introduce inductive biases and affect the learned representations. - The multi - stage pre - training process is complex, and usually different pre - training strategies are designed for VSR and ASR respectively, increasing the implementation difficulty. 2. **Research objectives**: - Propose a single - stage self - supervised method that can directly jointly learn visual and auditory speech representations from original video and audio data. - Through this method, achieve VSR and ASR performance superior to existing self - supervised methods with a small amount of labeled data. - Verify whether the proposed method can further improve performance on large - scale labeled datasets. ### Solutions proposed in the paper: - **Method name**: RA VEn (Raw Audio - Visual Speech Encoders) - **Core ideas**: - Use a pair of student - teacher networks to process data in visual and auditory modalities respectively. - The student network encodes the masked input and predicts the output of the unmasked teacher network through a lightweight Transformer predictor. - Since audio contains more speech information than video, the audio student network needs to predict the outputs from the video and audio teacher networks (cross - modal and intra - modal learning), while the video student network only predicts the output of the audio teacher network (cross - modal learning). ### Experimental results: - **Low - resource labeled data setting**: - When using only 30 hours of labeled data, RA VEn significantly outperforms the existing self - supervised methods and further improves performance through self - training. - On the LRS3 dataset, RA VEn achieves a 23.8% WER (Word Error Rate). - **High - resource labeled data setting**: - When fine - tuning using the complete 433 - hour LRS3 dataset, RA VEn still performs excellently, especially achieving a 1.4% WER in the ASR task. - Compared with the method trained with 90,000 hours of non - public data, RA VEn performs better in the low - resource setting. ### Conclusion: The paper shows that through single - stage self - supervised learning, visual and auditory speech representations can be effectively learned from original video and audio data, thereby significantly improving the performance of VSR and ASR in low - resource and high - resource labeled data settings.

Jointly Learning Visual and Auditory Speech Representations from Raw Data

BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision

Learning Cross-lingual Visual Speech Representations

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations

Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs

Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Self-supervised Learning of Audio Representations from Audio-Visual Data using Spatial Alignment

Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization

Self-Supervised Audio-Visual Speech Representations Learning by Multimodal Self-Distillation

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language

Leveraging Visual Supervision for Array-based Active Speaker Detection and Localization

Audiovisual Masked Autoencoders

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

Unified Cross-Modal Attention: Robust Audio-Visual Speech Recognition and Beyond

For end-to-end audio-visual speech recognition