Mohammad Taha ToghaniSoomin LeeCésar A. Uribea Department of Electrical and Computer Engineering,Rice University,Houston,TX,USAb Amazon,Haifa,IsraelMohammad Taha Toghani received the B.Sc. degree in electrical engineering and computer science from the Sharif University of Technology,Tehran,Iran,in 2019. He is currently working toward the Ph.D. degree in electrical engineering with the Department of Electrical and Computer Engineering,Rice University,Houston,TX,USA. His research interests span distributed optimization and large-scale machine learning.Soomin Lee received the Ph.D. degree in electrical and computer engineering from the University of Illinois at Urbana-Champaign,and two master's degrees in electrical engineering from the Korea Advanced Institute of Science and Technology (KAIST) and in computer science from the University of Illinois at Urbana-Champaign. After graduation,she worked as a Postdoc Associate in Duke Robotics Group and Industrial and Systems Engineering at Georgia Tech and she was a Senior Research Scientist at Yahoo! Research until 2022. She is currently a Senior Applied Scientist at Amazon. She is a recipient of NSF South Data Hub PEPI fellowship. Her research interests include theoretical optimization,control and optimization of distributed systems interconnected over complex networks,deep learning,and large-scale machine learning algorithms for big data analysis.César A. Uribe received the M.Sc. degree in systems and control from the Delft University of Technology,Delft,The Netherlands,in 2013,and the M.Sc. degree in applied mathematics and the Ph.D. degree in electrical and computer engineering from the University of Illinois at Urbana-Champaign,Urbana-Champaign,IL,USA,in 2016 and 2018,respectively. He is currently the Louis Owen Jr. Assistant Professor with the Department of Electrical and Computer Engineering,Rice University,Houston,TX,USA. He was a Postdoctoral Associate with the Laboratory for Information and Decision Systems (LIDS),Massachusetts Institute of Technology (MIT),Cambridge,MA,USA,until 2020 and held a visiting Professor position at the Moscow Institute of Physics and Technology,Dolgoprudny,Russia. His research interests include distributed learning and optimization,decentralized control,algorithm analysis,and computational optimal transport.

Abstract:We study the personalized federated learning problem under asynchronous updates. In this problem, each client seeks to obtain a personalized model that simultaneously outperforms local and global models. We consider two optimization-based frameworks for personalization: (i) Model-Agnostic Meta-Learning ( MAML ) and (ii) Moreau Envelope ( ME ). MAML involves learning a joint model adapted for each client through fine-tuning, whereas ME requires a bi-level optimization problem with implicit gradients to enforce personalization via regularized losses. We focus on improving the scalability of personalized federated learning by removing the synchronous communication assumption. Moreover, we extend the studied function class by removing boundedness assumptions on the gradient norm. Our main technical contribution is a unified proof for asynchronous federated learning with bounded staleness that we apply to MAML and ME personalization frameworks. For the smooth and non-convex functions class, we show the convergence of our method to a first-order stationary point. We illustrate the performance of our method and its tolerance to staleness through experiments for classification tasks over heterogeneous datasets.

PersA-FL: Personalized Asynchronous Federated Learning

PersA-FL: personalized asynchronous federated learning

Personalized Federated Learning: A Meta-Learning Approach

AsyncFedED: Asynchronous Federated Learning with Euclidean Distance Based Adaptive Weight Aggregation

Improving Federated Learning Personalization via Model Agnostic Meta Learning

Personalized Federated Learning with First Order Model Optimization

FedL2P: Federated Learning to Personalize

Personalized Federated Learning via Amortized Bayesian Meta-Learning

Personalized Federated Learning with Moreau Envelopes

Personalized Federated Learning through Local Memorization

Personalized Federated Learning with Mixture of Models for Adaptive Prediction and Model Fine-Tuning

Personalized Federated Learning with Local Optimization Models

Federated Learning with Partial Model Personalization

Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques

Collaborative and Efficient Personalization with Mixtures of Adaptors

Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning

Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity

Sparse Personalized Federated Learning

Personalized Federated Learning: A Clustered Distributed Co-Meta-Learning Approach

Learn What You Need in Personalized Federated Learning