Abstract:Multimodal learning methods with targeted unimodal learning objectives have exhibited their superior efficacy in alleviating the imbalanced multimodal learning problem. However, in this paper, we identify the previously ignored gradient conflict between multimodal and unimodal learning objectives, potentially misleading the unimodal encoder optimization. To well diminish these conflicts, we observe the discrepancy between multimodal loss and unimodal loss, where both gradient magnitude and covariance of the easier-to-learn multimodal loss are smaller than the unimodal one. With this property, we analyze Pareto integration under our multimodal scenario and propose MMPareto algorithm, which could ensure a final gradient with direction that is common to all learning objectives and enhanced magnitude to improve generalization, providing innocent unimodal assistance. Finally, experiments across multiple types of modalities and frameworks with dense cross-modal interaction indicate our superior and extendable method performance. Our method is also expected to facilitate multi-task cases with a clear discrepancy in task difficulty, demonstrating its ideal scalability. The source code and dataset are available at <a class="link-external link-https" href="https://github.com/GeWu-Lab/MMPareto_ICML2024" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

This paper mainly discusses the problem of gradient conflict between multi-modal and single-modal learning objectives in multi-modal learning. This conflict may mislead the optimization of single-modal encoders and affect the performance of the model. The authors observed that the gradient magnitude and covariance of multi-modal losses are usually smaller than those of single-modal losses, which provides insights for resolving the conflict. The paper proposes the MMPareto algorithm, which aims to provide innocent single-modal assistance by analyzing the reasons for the failure of Pareto optimization in multi-modal scenarios, ensuring that the final gradient direction is consistent with all learning objectives, and enhancing the magnitude to improve generalization ability. The MMPareto algorithm considers the direction and magnitude of gradients, and performs gradient integration separately in conflicting and non-conflicting situations to avoid optimization conflicts and enhance the generalization ability of the model. Experimental results show that the MMPareto method exhibits superior and scalable performance in various types of datasets and frameworks with dense cross-modal interactions, addressing the imbalance issue in multi-modal learning. Moreover, this method can also be applied to multi-task scenarios with significant difficulty differences, demonstrating good scalability.

MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance