Multimodal Contrastive Training for Visual Representation Learning

Xin Yuan,Zhe Lin,Jason Kuen,Jianming Zhang,Yilin Wang,Michael Maire,Ajinkya Kale,Baldo Faieta

DOI: https://doi.org/10.48550/arXiv.2104.12836

2021-04-27

Abstract:We develop an approach to learning visual representations that embraces multimodal data, driven by a combination of intra- and inter-modal similarity preservation objectives. Unlike existing visual pre-training methods, which solve a proxy prediction task in a single domain, our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations. By including multimodal training in a unified framework with different types of contrastive losses, our method can learn more powerful and generic visual features. We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation. For example, the visual representations pre-trained on COCO by our method achieve state-of-the-art top-1 validation accuracy of $55.3\%$ on ImageNet classification, under the common transfer protocol. We also evaluate our method on the large-scale Stock images dataset and show its effectiveness on multi-label image tagging, and cross-modal retrieval tasks.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to learn high - quality visual representations from multi - modal data. Specifically, the paper proposes a new method. This method improves the quality of the learned visual representations by combining intra - modal and inter - modal similarity - preserving objectives, leveraging the data characteristics of different modalities under a unified training framework, and the semantic information in cross - modal correlations. Compared with the existing single - domain proxy prediction tasks, this method can more fully mine the intrinsic data characteristics of each modality and the cross - modal correlations, generating more powerful and general visual features.

The main contributions of the paper are as follows:
1. **Unified multi - modal training framework**: The proposed framework can self - supervisedly fully utilize the data potential in each unlabeled modality and bridge the heterogeneity gap by comparing different modalities in a common semantic space. In addition, the framework is flexible and can be easily extended to incorporate any emerging modality.
2. **Extensive transferability**: The visual representations pre - trained by this method can be transferred in many downstream computer vision tasks and perform excellently under common transfer learning protocols. Experimental results show that this method has achieved excellent performance on multiple tasks such as image classification, object detection, and instance segmentation.

Through these improvements, the paper aims to overcome the problem that existing self - supervised learning methods lack high - level conceptual semantic information, while reducing the dependence on a large amount of labeled data, providing an efficient and powerful visual representation learning method.