Abstract:Benefiting from masked visual modeling, self-supervised video representation learning has achieved remarkable progress. However, existing methods focus on learning representations from scratch through reconstructing low-level features like raw pixel values. In this paper, we propose masked video distillation (MVD), a simple yet effective two-stage masked feature modeling framework for video representation learning: firstly we pretrain an image (or video) model by recovering low-level features of masked patches, then we use the resulting features as targets for masked feature modeling. For the choice of teacher models, we observe that students taught by video teachers perform better on temporally-heavy video tasks, while image teachers transfer stronger spatial representations for spatially-heavy video tasks. Visualization analysis also indicates different teachers produce different learned patterns for students. To leverage the advantage of different teachers, we design a spatial-temporal co-teaching method for MVD. Specifically, we distill student models from both video teachers and image teachers by masked feature modeling. Extensive experimental results demonstrate that video transformers pretrained with spatial-temporal co-teaching outperform models distilled with a single teacher on a multitude of video datasets. Our MVD with vanilla ViT achieves state-of-the-art performance compared with previous methods on several challenging video downstream tasks. For example, with the ViT-Large model, our MVD achieves 86.4% and 76.7% Top-1 accuracy on Kinetics-400 and Something-Something-v2, outperforming VideoMAE by 1.2% and 2.4% respectively. When a larger ViT-Huge model is adopted, MVD achieves the state-of-the-art performance with 77.3% Top-1 accuracy on Something-Something-v2. Code will be available at https://github.com/ruiwang2021/mvd.

Self-Supervised Audio-Visual Speech Representations Learning by Multimodal Self-Distillation

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

Audio-Visual Speech Enhancement with Deep Multi-modality Fusion

AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition

Audio-Visual Speech Separation with Visual Features Enhanced by Adversarial Training

Self-Supervised Audio-Visual Speaker Representation with Co-Meta Learning

Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition

Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification

Multimodality Self-distillation for Fast Inference of Vision and Language Pretrained Models

A Comprehensive Study on Self-Supervised Distillation for Speaker Representation Learning

VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning

Self-Distillation Prototypes Network: Learning Robust Speaker Representations without Supervision

Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection

Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning.

Jointly Learning Visual and Auditory Speech Representations from Raw Data

Deep Multimodal Learning for Audio-Visual Speech Recognition

token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text