Abstract:Multimodal learning often outperforms its unimodal counterparts by exploiting unimodal contributions and cross-modal interactions. However, focusing only on integrating multimodal features into a unified comprehensive representation overlooks the unimodal characteristics. In real data, the contributions of modalities can vary from instance to instance, and they often reinforce or conflict with each other. In this study, we introduce a novel \text{MultiModal} loss paradigm for multimodal learning, which subgroups instances according to their unimodal contributions. \text{MultiModal} loss can prevent inefficient learning caused by overfitting and efficiently optimize multimodal models. On synthetic data, \text{MultiModal} loss demonstrates improved classification performance by subgrouping difficult instances within certain modalities. On four real multimodal datasets, our loss is empirically shown to improve the performance of recent models. Ablation studies verify the effectiveness of our loss. Additionally, we show that our loss generates a reliable prediction score for each modality, which is essential for subgrouping. Our \text{MultiModal} loss is a novel loss function to subgroup instances according to the contribution of modalities in multimodal learning and is applicable to a variety of multimodal models with unimodal decisions. Our code is available at <a class="link-external link-https" href="https://github.com/SehwanMoon/MultiModalLoss" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
This paper attempts to solve the problem in multimodal learning that existing methods fail to fully consider the contribution of different modalities to instances. Specifically, existing multimodal learning methods usually generate a unified representation by integrating information from different modalities. However, this method often ignores the unique characteristics of each modality. Moreover, in actual data, the contribution of different modalities to instances may vary from instance to instance, and may even conflict or reinforce each other. Therefore, this simple integration method may lead to model over - fitting or ineffective optimization.
To solve the above problems, the author proposes a new multimodal loss function (MultiModal loss), which can group instances according to the contribution of each modality to the instance. In this way, MultiModal loss can prevent the low learning efficiency caused by over - fitting and effectively optimize the multimodal model. In addition, this loss function can also generate reliable prediction scores for each modality, which is crucial for the grouping operation.
### Main contributions:
1. **Propose a new multimodal loss function**: This loss function can group instances according to modal contributions and is applicable to a variety of multimodal models.
2. **Verification on synthetic data**: By generating synthetic data containing difficult instances in certain modalities, it is shown that existing methods ignoring these difficult modalities will lead to performance degradation, while MultiModal loss can perform well without ignoring these difficult modalities.
3. **Application on real - world datasets**: The proposed loss function is applied to four real - world multimodal datasets, demonstrating its performance improvement.
4. **Analysis and verification of design decisions**: Through the analysis of the design decisions of the loss function, its effectiveness and rationality are further verified.
### Technical details:
- **Definition of multimodal loss function**: MultiModal loss is a proxy - based loss function. It performs instance grouping by calculating the similarity scores between queries (outputs of different modalities) and all keys (proxies). The specific formula is as follows:
\[
\text{sim}_{i,c}^k=\sum_m \mathbf{x}_m^i\cdot\mathbf{w}_{m,c}^k
\]
where $\mathbf{x}_m^i$ represents the output of the $i$-th instance in the $m$-th modality, and $\mathbf{w}_{m,c}^k$ represents the $k$-th proxy of the $c$-th class.
- **Attention mechanism**: To ensure the reliability of the output of each modality, a soft - attention mechanism is used to calculate the attention weight of each modality:
\[
\text{att}_{i,c}^k = \frac{\exp\left(\frac{1}{\gamma}\text{sim}_{i,c}^k\right)}{\sum_{c'}\sum_{k'}\exp\left(\frac{1}{\gamma}\text{sim}_{i,c'}^{k'}\right)}
\]
where $\gamma$ is a scaling factor.
- **Normalized similarity score**: To better reflect the characteristics of each modality, the similarity score is normalized:
\[
S_{i,c}=\sum_k\frac{\exp\left(\frac{1}{\gamma}\text{sim}_{i,c}^k\right)}{\sum_{c'}\exp\left(\frac{1}{\gamma}\text{sim}_{i,c'}^k\right)}\cdot\text{sim}_{i,c}^k
\]
- **Multimodal loss function**: The final multimodal loss function is defined as:
\[
\ell_{\text{MultiModal}}=-\log\frac{\exp(S_{i,y_i}+A_{i,y_i})}{\sum_c\exp(S_{i,c}+A_{i,c})}
\]