Abstract:Fully decentralized learning is gaining momentum for training AI models at the Internet's edge, addressing infrastructure challenges and privacy concerns. In a decentralized machine learning system, data is distributed across multiple nodes, with each node training a local model based on its respective dataset. The local models are then shared and combined to form a global model capable of making accurate predictions on new data. Our exploration focuses on how different types of network structures influence the spreading of knowledge - the process by which nodes incorporate insights gained from learning patterns in data available on other nodes across the network. Specifically, this study investigates the intricate interplay between network structure and learning performance using three network topologies and six data distribution methods. These methods consider different vertex properties, including degree centrality, betweenness centrality, and clustering coefficient, along with whether nodes exhibit high or low values of these metrics. Our findings underscore the significance of global centrality metrics (degree, betweenness) in correlating with learning performance, while local clustering proves less predictive. We highlight the challenges in transferring knowledge from peripheral to central nodes, attributed to a dilution effect during model aggregation. Additionally, we observe that central nodes exert a pull effect, facilitating the spread of knowledge. In examining degree distribution, hubs in Barabasi-Albert networks positively impact learning for central nodes but exacerbate dilution when knowledge originates from peripheral nodes. Finally, we demonstrate the formidable challenge of knowledge circulation outside of segregated communities.

An Overview of Autonomous Connection Establishment Methods in Peer-to-Peer Deep Learning

Peer-to-Peer Learning + Consensus with Non-IID Data

Peer-to-Peer Learning Dynamics of Wide Neural Networks

A Graph Neural Network Based Decentralized Learning Scheme

MAPL: Model Agnostic Peer-to-peer Learning

Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs

Initialisation and Network Effects in Decentralised Federated Learning

Dominating Set Model Aggregation for communication-efficient decentralized deep learning

Decentralized Deep Learning using Momentum-Accelerated Consensus

How to Organize your Deep Reinforcement Learning Agents: The Importance of Communication Topology

Improving Unstructured Peer-to-peer Systems by Adaptive Connection Establishment

Peer-to-Peer Deep Learning for Beyond-5G IoT

Decentralized Machine Learning Through Experience-Driven Method in Edge Networks

Learning-Driven Decentralized Machine Learning in Resource-Constrained Wireless Edge Computing.

The effect of network topologies on fully decentralized learning: a preliminary investigation

Impact of network topology on the performance of Decentralized Federated Learning

RNEP: Random Node Entropy Pairing for Efficient Decentralized Training with Non-IID Local Data

Personalized and Private Peer-to-Peer Machine Learning

Personalized Decentralized Multi-Task Learning Over Dynamic Communication Graphs

Decentralized Learning Made Easy with DecentralizePy

Advances of Pipeline Model Parallelism for Deep Learning Training: An Overview