Abstract:“Big data” is always collected from different resources that have different data structures. With the rapid development of information technologies, current precious data resources are characteristic of multimodes. As a result, based on classical machine learning strategies, multi-modal learning has become a valuable research topic, enabling computers to process and understand “big data”. The cognitive processes of humans involve perception through different sense organs. Signals from eyes, ears, the nose, and hands (tactile sense) constitute a person’s understanding of a special scene or the world as a whole. It reasonable to believe that multi-modal methods involving a higher ability to process complex heterogeneous data can further promote the progress of information technologies. The concepts of multimodality stemmed from psychology and pedagogy from hundreds of years ago and have been popular in computer science during the past decade. In contrast to the concept of “media”, a “mode” is a more fine-grained concept that is associated with a typical data source or data form. The effective utilization of multi-modal data can aid a computer understand a specific environment in a more holistic way. In this context, we first introduced the definition and main tasks of multi-modal learning. Based on this information, the mechanism and origin of multi-modal machine learning were then briefly introduced. Subsequently, statistical learning methods and deep learning methods for multi-modal tasks were comprehensively summarized. We also introduced the main styles of data fusion in multi-modal perception tasks, including feature representation, shared mapping, and co-training. Additionally, novel adversarial learning strategies for cross-modal matching or generation were reviewed. The main methods for multi-modal learning were outlined in this paper with a focus on future research issues in this field.

Multimodality in meta-learning: A comprehensive survey

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Revisit Multimodal Meta-Learning through the Lens of Multi-Task Learning

A Survey of Meta-learning for Classification Tasks

Cross-Modal Meta-Knowledge Transfer: A Meta-Learning Framework Adaptable for Multimodal Tasks

A Survey of Multimodal Machine Learning

Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

VL-Meta: Vision-Language Models for Multimodal Meta-Learning

Recent Advances of Multimodal Continual Learning: A Comprehensive Survey

Multimodal meta-learning through meta-learned task representations

Learning to Learn Better Unimodal Representations Via Adaptive Multimodal Meta-Learning

Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning

A survey of multimodal federated learning: background, applications, and perspectives

Multimodal Machine Learning: A Survey and Taxonomy

Surveying the MLLM Landscape: A Meta-Review of Current Surveys

A Survey of Multimodal Large Language Model from A Data-centric Perspective

Deep Multimodal Learning with Missing Modality: A Survey

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications