Abstract:In classification tasks, such as face recognition and emotion recognition, multimodal information is used for accurate classification. Once a multimodal classification model is trained with a set of modalities, it estimates the class label by using the entire modality set. A trained classifier is typically not formulated to perform classification for various subsets of modalities. Thus, the model would be useful and portable if it could be used for any subset of modalities. We refer to this problem as the multimodal portability problem. Moreover, in the multimodal model, classification accuracy is reduced when one or more modalities are missing. We term this problem the missing modality problem. This article proposes a novel deep learning model, termed KModNet, and a novel learning strategy, termed progressive learning, to simultaneously address missing modality and multimodal portability problems. KModNet, formulated with the transformer, contains multiple branches corresponding to different k-combinations of the modality set S. KModNet is trained using a multi-step progressive learning framework, where the k-th step uses a k-modal model to train different branches up to the k-th combination branch. To address the missing modality problem, the training multimodal data is randomly ablated. The proposed learning framework is formulated and validated using two multimodal classification problems: audio-video-thermal person classification and audio-video emotion classification. The two classification problems are validated using the Speaking Faces, RAVDESS, and SAVEE datasets. The results demonstrate that the progressive learning framework enhances the robustness of multimodal classification, even under the conditions of missing modalities, while being portable to different modality subsets.

Training Multimodal Systems for Classification with Multiple Objectives

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Multimodal Learning for View-Based 3D Object Classification

Progressive Learning of a Multimodal Classifier Accounting for Different Modality Combinations

Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks

Improving Unimodal Inference with Multimodal Transformers

Learn to Combine Modalities in Multimodal Deep Learning

Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach

On-the-fly Modulation for Balanced Multimodal Learning

Improving Multimodal Learning with Multi-Loss Gradient Modulation

Improving Multi-Modal Learning with Uni-Modal Teachers

Brain-inspired Multimodal Learning Based on Neural Networks

Multimodal Representation Learning by Alternating Unimodal Adaptation

A Theory of Multimodal Learning

Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Balanced Multimodal Learning via On-the-fly Gradient Modulation

Deep Multi-Modal Sets

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

One-stage Modality Distillation for Incomplete Multimodal Learning