Abstract:With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: <a class="link-external link-https" href="https://github.com/wangxiao5791509/MultiModal_BigModels_Survey" rel="external noopener nofollow">this https URL</a>. This paper has been published by the journal Machine Intelligence Research (MIR), <a class="link-external link-https" href="https://link.springer.com/article/10.1007/s11633-022-1410-8" rel="external noopener nofollow">this https URL</a>, DOI: <a class="link-https link-external" data-doi="10.1007/s11633-022-1410-8" href="https://doi.org/10.1007/s11633-022-1410-8" rel="external noopener nofollow">https://doi.org/10.1007/s11633-022-1410-8</a>, vol. 20, no. 4, pp. 447-482, 2023.

MCHPT: A Weakly Supervise Based Merchant Pre-trained Model.

Knowledge Perceived Multi-modal Pretraining in E-commerce

Short Text Pre-training with Extended Token Classification for E-commerce Query Understanding

Multimodal Pretraining from Monolingual to Multilingual

Retrieval Oriented Masking Pre-training Language Model for Dense Passage Retrieval

BERTifying the Hidden Markov Model for Multi-Source Weakly Supervised Named Entity Recognition

PTUM: Pre-training User Model from Unlabeled User Behaviors Via Self-supervision.

CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation

Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-training

POS-BERT: Point Cloud One-Stage BERT Pre-Training

MPNet: Masked and Permuted Pre-training for Language Understanding

CharBERT: Character-aware Pre-trained Language Model

CPM-2: Large-scale Cost-effective Pre-trained Language Models

Pre-training Language Model As a Multi-perspective Course Learner.

Sentiment-aware Multimodal Pre-Training for Multimodal Sentiment Analysis

mPMR: A Multilingual Pre-trained Machine Reader at Scale

Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation

Boosting Point-BERT by Multi-choice Tokens

From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Reader

Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey