BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translation

Liyan Kang,Luyang Huang,Ningxin Peng,Peihao Zhu,Zewei Sun,Shanbo Cheng,Mingxuan Wang,Degen Huang,Jinsong Su

2023-07-03

Abstract:We present a large-scale video subtitle translation dataset, BigVideo, to facilitate the study of multi-modality machine translation. Compared with the widely used How2 and VaTeX datasets, BigVideo is more than 10 times larger, consisting of 4.5 million sentence pairs and 9,981 hours of videos. We also introduce two deliberately designed test sets to verify the necessity of visual information: Ambiguous with the presence of ambiguous words, and Unambiguous in which the text context is self-contained for translation. To better model the common semantics shared across texts and videos, we introduce a contrastive learning method in the cross-modal encoder. Extensive experiments on the BigVideo show that: a) Visual information consistently improves the NMT model in terms of BLEU, BLEURT, and COMET on both Ambiguous and Unambiguous test sets. b) Visual information helps disambiguation, compared to the strong text baseline on terminology-targeted scores and human evaluation. Dataset and our implementations are available at <a class="link-external link-https" href="https://github.com/DeepLearnXMU/BigVideo-VMT" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address the effective utilization of video information in Multimodal Machine Translation (MMT). Specifically, the authors propose a large-scale video subtitle translation dataset **BIGVIDEO** to promote research in multimodal machine translation. Compared to existing datasets such as **HOW2** and **VATEX**, **BIGVIDEO** is larger in scale, containing 4.5 million sentence pairs and 9,981 hours of video.

### Main Contributions

1. **Dataset Construction**:
- **Scale**: **BIGVIDEO** is more than ten times larger than the largest existing video translation dataset, containing 155,000 videos and 4.5 million high-quality parallel sentences.
- **Diversity**: The dataset covers all categories of videos from YouTube and Xigua platforms, ensuring data diversity and complexity.
- **Test Set Design**: Two carefully designed test sets, **AMBIGUOUS** and **UNAMBIGUOUS**, are introduced to validate the necessity of visual information in handling semantic ambiguity and self-contained text.

2. **Model Improvements**:
- **Unified Encoder-Decoder Framework**: A unified encoder-decoder framework is proposed to handle both video and text inputs simultaneously.
- **Contrastive Learning Method**: A contrastive learning objective is introduced to further bridge the gap between text and video representations, enhancing the role of visual information in the translation model.

### Experimental Results

1. **Performance Improvement**:
- On the **BIGVIDEO** dataset, experimental results show that visual information significantly improves the performance of NMT models on both **AMBIGUOUS** and **UNAMBIGUOUS** test sets, especially in BLEU, BLEURT, and COMET metrics.
- The contrastive learning method further enhances the model's translation performance, particularly in term-specific metrics.

2. **Disambiguation Capability**:
- Visual information helps resolve semantic ambiguity issues, excelling in metrics such as term accuracy, window overlap, and term bias translation edit rate (1-TERm).

### Conclusion

This study demonstrates the importance and effectiveness of visual information in multimodal machine translation by constructing a large-scale and diverse **BIGVIDEO** dataset. The introduction of the contrastive learning method further improves model performance, providing new directions and tools for future research.