Abstract:Federated recommendation systems are essential for providing personalized recommendations while protecting user privacy. However, current methods mainly rely on ID-based item embeddings, neglecting the rich multimodal information of items. To address this, we propose a Federated Multimodal Recommendation System, called FedMR. FedMR uses a foundation model on the server to encode multimodal item data, such as images and text. To handle data heterogeneity caused by user preference differences, FedMR introduces a Mixing Feature Fusion Module on each client, which adjusts fusion strategy weights based on user interaction history to generate personalized item representations that capture users' fine-grained preferences. FedMR is compatible with existing ID-based federated recommendation systems, improving performance without modifying the original framework. Experiments on four real-world multimodal datasets demonstrate FedMR's effectiveness. The code is available at <a class="link-external link-https" href="https://anonymous.4open.science/r/FedMR" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem of how to use multimodal data to generate personalized item representations in the federated recommendation system while ensuring user privacy. Specifically, the current federated recommendation systems (FedRec) mainly rely on ID - based item embeddings and ignore the rich multimodal information of items (such as images and text). This leads to limitations in recommendation performance, especially in dealing with data sparsity and differences in user preferences. #### Main problems: 1. **Data sparsity**: Each client can usually only access the interaction history of a single user, which leads to the data sparsity problem and thus reduces the recommendation performance. 2. **Insufficient use of multimodal information**: Although items usually contain rich multimodal information (such as text and images), most existing FedRec methods still mainly rely on ID embeddings and fail to fully utilize the potential of multimodal data. 3. **Challenges in personalized representation**: Generating personalized item representations that capture the fine - grained preferences of users while ensuring user privacy is an important challenge. ### Solutions: To address these challenges, the authors propose a new framework named FedMR (Federated Multimodal Recommendation System). The main contributions of FedMR are as follows: 1. **Fusing multimodal information**: FedMR uses the server - side Foundation Model (FM) to encode multimodal item data (such as images and text) to generate feature embeddings. 2. **Dynamically adjusting the fusion strategy**: A Mixing Feature Fusion Module (MFFM) is introduced at each client to dynamically adjust the weights of the fusion strategy according to the user's interaction history, in order to generate personalized item representations. 3. **Compatibility with existing frameworks**: FedMR is compatible with existing ID - based federated recommendation systems and can improve the recommendation performance without modifying the original framework. 4. **Experimental verification**: Extensive experiments were carried out on four real - world multimodal datasets to verify the effectiveness of FedMR. ### Summary: The core problem of this paper is to solve how to effectively use multimodal data to generate personalized item representations in the federated recommendation system under the premise of protecting user privacy, so as to improve the performance of the recommendation system. FedMR successfully solves this problem by introducing the foundation model and the mixing feature fusion module, and shows its superior performance on multiple datasets.

Personalized Item Representations in Federated Multimodal Recommendation

Dual Personalization on Federated Recommendation

Beyond Similarity: Personalized Federated Recommendation with Composite Aggregation

Robust Privacy-Preserving Recommendation Systems Driven by Multimodal Federated Learning

Federated Recommendation with Additive Personalization

Towards Communication Efficient and Fair Federated Personalized Sequential Recommendation

Personalized Federated Recommendation via Joint Representation Learning, User Clustering, and Model Adaptation

Practical and Secure Federated Recommendation with Personalized Mask

FMMRec: Fairness-aware Multimodal Recommendation

Efficient and Robust Regularized Federated Recommendation

Practical and Secure Federated Recommendation with Personalized Masks

GPFedRec: Graph-guided Personalization for Federated Recommendation

Federated Recommendation via Hybrid Retrieval Augmented Generation

PerFedRec++: Enhancing Personalized Federated Recommendation with Self-Supervised Pre-Training

Train Once, Deploy Anywhere: Matryoshka Representation Learning for Multimodal Recommendation

Discrete Federated Multi-behavior Recommendation for Privacy-Preserving Heterogeneous One-Class Collaborative Filtering

ID Embedding as Subtle Features of Content and Structure for Multimodal Recommendation

HeteFedRec: Federated Recommender Systems with Model Heterogeneity

When Federated Recommendation Meets Cold-Start Problem: Separating Item Attributes and User Interactions

FedRec: Federated Recommendation With Explicit Feedback

Cluster-driven Personalized Federated Recommendation with Interest-aware Graph Convolution Network for Multimedia