Abstract:Training supervised video captioning model requires coupled video-caption pairs. However, for many targeted languages, sufficient paired data are not available. To this end, we introduce the unpaired video captioning task aiming to train models without coupled video-caption pairs in target language. To solve the task, a natural choice is to employ a two-step pipeline system: first utilizing video-to-pivot captioning model to generate captions in pivot language and then utilizing pivot-to-target translation model to translate the pivot captions to the target language. However, in such a pipeline system, 1) visual information cannot reach the translation model, generating visual irrelevant target captions; 2) the errors in the generated pivot captions will be propagated to the translation model, resulting in disfluent target captions. To address these problems, we propose the Unpaired Video Captioning with Visual Injection system (UVC-VI). UVC-VI first introduces the Visual Injection Module (VIM), which aligns source visual and target language domains to inject the source visual information into the target language domain. Meanwhile, VIM directly connects the encoder of the video-to-pivot model and the decoder of the pivot-to-target model, allowing end-to-end inference by completely skipping the generation of pivot captions. To enhance the cross-modality injection of the VIM, UVC-VI further introduces a pluggable video encoder, i.e., Multimodal Collaborative Encoder (MCE). The experiments show that UVC-VI outperforms pipeline systems and exceeds several supervised systems. Furthermore, equipping existing supervised systems with our MCE can achieve 4% and 7% relative margins on the CIDEr scores to current state-of-the-art models on the benchmark MSVD and MSR-VTT datasets, respectively.

Unsupervised Multimodal Video-to-Video Translation via Self-Supervised Learning

Multimodal Image-to-Image Translation via Mutual Information Estimation and Maximization

An Unsupervised Video Summarization Method Based on Multimodal Representation.

UATST: Towards Unpaired Arbitrary Text-Guided Style Transfer with Cross-Space Modulation

Unsupervised Teacher-Student Model for Large-Scale Video Retrieval.

Video Pivoting Unsupervised Multi-Modal Machine Translation

Unsupervised content and style learning for multimodal cross-domain image translation

Unsupervised Multimodal Image-to-Image Translation: Generate What You Want.

Temporally Consistent Unpaired Multi-domain Video Translation by Contrastive Learning

UVIS: Unsupervised Video Instance Segmentation

Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination

Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models

Visual Grounding in Video for Unsupervised Word Translation

Aligning Source Visual and Target Language Domains for Unpaired Video Captioning

UniVST: A Unified Framework for Training-free Localized Video Style Transfer

Unsupervised Multi-Domain Multimodal Image-to-image Translation with Explicit Domain-Constrained Disentanglement.

Video-to-Video Translation with Global Temporal Consistency.

Self-Supervised Adversarial Video Summarizer with Context Latent Sequence Learning.

Jointly Modeling Embedding and Translation to Bridge Video and Language

Unsupervised Variational Translator for Bridging Image Restoration and High-Level Vision Tasks

UVCGAN: UNet Vision Transformer cycle-consistent GAN for unpaired image-to-image translation