Abstract:With the rapid growth of multimedia-sharing platforms (e.g. Twitter and TikTok), multimedia recommender systems have become fundamental for helping users alleviate information overload and discover items of interest. Existing multimedia recommendation methods often incorporate various auxiliary modalities (e.g., visual, textual, and acoustic) to describe item characteristics and improve task performance. However, these methods usually assume that each item is associated with complete modalities, ignoring the prevalence of missing modality issues in real-world scenarios. To deal with the challenge of missing modalities, in this paper, we propose a novel framework of Contrastive Intra- and Inter-Modality Generation (CI2MG) for enhancing incomplete multimedia recommendation. We first develop a contrastive intra- and inter-modality generation module for the missing modalities, where the intra-modality representation is updated through clustering-based hypergraph convolution and inter-modality representation is obtained by optimal transport between different modalities. To tackle the challenge of insufficient and incomplete supervision labels during intra- and inter-modality generation, a modality-aware contrastive learning paradigm is introduced based on an augmentation between the intra-modality view and inter-modality view. Furthermore, to learn task-related representations from the generative modalities and further improve the performance of recommendation, we design an enhanced multimedia recommendation module to alleviate the influences driven by task-irrelevant noise. Extensive experiments on real-world datasets show the superiority of our proposed CI2MG framework in offering great potential for personalized multimedia recommendation over the state-of-the-art baselines regarding Recall, NDCG, and Precision metrics.

Intra- and Inter-Modal Curriculum for Multimodal Learning

Contrastive Intra- and Inter-Modality Generation for Enhancing Incomplete Multimedia Recommendation

Multimodal Classification via Modal-Aware Interactive Enhancement

One-stage Modality Distillation for Incomplete Multimodal Learning

Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning

On-the-fly Modulation for Balanced Multimodal Learning

Diagnosing and Re-learning for Balanced Multimodal Learning

Explaining and Mitigating the Modality Gap in Contrastive Multimodal Learning

Balanced Multimodal Learning via On-the-fly Gradient Modulation

What Makes Multimodal In-Context Learning Work?

Multimodal Fusion Balancing Through Game-Theoretic Regularization

What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

Detached and Interactive Multimodal Learning

Improving Multi-Modal Learning with Uni-Modal Teachers

From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning

Multimodal Understanding Through Correlation Maximization and Minimization

MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance

Towards Multimodal In-Context Learning for Vision & Language Models

Learning Unseen Modality Interaction

Towards Balanced Active Learning for Multimodal Classification

PMR: Prototypical Modal Rebalance for Multimodal Learning