Abstract:Distributed multi-task learning provides significant advantages in multi-agent networks with heterogeneous data sources where agents aim to learn distinct but correlated models <a class="link-external link-http" href="http://simultaneously.However" rel="external noopener nofollow">this http URL</a>, distributed algorithms for learning relatedness among tasks are not resilient in the presence of Byzantine agents. In this paper, we present an approach for Byzantine resilient distributed multi-task learning. We propose an efficient online weight assignment rule by measuring the accumulated loss using an agent's data and its neighbors' models. A small accumulated loss indicates a large similarity between the two tasks. In order to ensure the Byzantine resilience of the aggregation at a normal agent, we introduce a step for filtering out larger losses. We analyze the approach for convex models and show that normal agents converge resiliently towards the global <a class="link-external link-http" href="http://minimum.Further" rel="external noopener nofollow">this http URL</a>, aggregation with the proposed weight assignment rule always results in an improved expected regret than the non-cooperative case. Finally, we demonstrate the approach using three case studies, including regression and classification problems, and show that our method exhibits good empirical performance for non-convex models, such as convolutional neural networks.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the lack of fault - tolerance in model aggregation in Distributed Multi - Task Learning (DMTL) in the presence of Byzantine nodes. Specifically: 1. **Problem Background**: - In distributed multi - task learning, multiple agents collaborate through the network to learn different but related task models. - Traditional methods evaluate similarity by measuring the distance between model parameters of two agents and adjust weights accordingly for model aggregation. - However, these methods are vulnerable to attacks in the presence of malicious or faulty Byzantine agents, causing normal agents to fail to converge to the global optimal solution. 2. **Specific Problems**: - Byzantine agents can send arbitrary information to other agents to disrupt the convergence performance of the network. - The weight - adjustment rules based on the distance of model parameters are easily exploited by Byzantine agents, causing normal agents to wrongly assign too high weights to malicious agents, thus affecting the learning effect. 3. **Solutions**: - The paper proposes an online weight - adjustment rule based on accumulated loss instead of relying on the distance between model parameters. - By introducing a filtering mechanism, normal agents will ignore neighbor information whose loss values are greater than their own, thereby reducing the influence of Byzantine agents. - This method ensures that in the presence of any number of Byzantine agents, normal agents can still robustly converge to the global optimal solution, and the cooperation effect is better than that without cooperation. 4. **Contributions**: - An efficient online weight - adjustment rule that is robust against Byzantine attacks is proposed. - It is proved that the result of aggregation using this rule is always better than the expected regret value in the case of non - cooperation. - Experimental results show that this method exhibits good performance in regression and classification problems, and even has a very good effect on non - convex models such as Convolutional Neural Networks (CNN). In summary, this paper aims to solve the robustness problem against Byzantine attacks in distributed multi - task learning, proposes an online weight - adjustment rule based on accumulated loss, and verifies its effectiveness and robustness through theoretical analysis and experimental verification.

Byzantine Resilient Distributed Multi-Task Learning

Distributed Learning of Predictive Structures from Multiple Tasks over Networks

Distributed Jointly Sparse Multitask Learning over Networks

Distributed Networked Multi-task Learning

Asynchronous Byzantine-Robust Stochastic Aggregation with Variance Reduction for Distributed Learning

Byzantine-Resilient Decentralized Collaborative Learning.

Distributed Multi-Task Relationship Learning

Adaptive Distributed Learning with Byzantine Robustness: A Gradient-Projection-Based Method

Online Parallel Multi-Task Relationship Learning via Alternating Direction Method of Multipliers

Byzantine-Robust and Communication-Efficient Distributed Learning via Compressed Momentum Filtering

Fault Tolerant ML: Efficient Meta-Aggregation and Synchronous Training

Efficient Byzantine-Resilient Stochastic Gradient Desce

Byzantine-Resilient Distributed Optimization of Multi-Dimensional Functions

Byzantine-Resilient Federated Learning at Edge

Distributed Stochastic Multi-Task Learning with Graph Regularization

Robust Task Grouping with Representative Tasks for Clustered Multi-Task Learning

Robust Distributed Learning Against Both Distributional Shifts and Byzantine Attacks

Distributed Multi-Task Learning with Shared Representation

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers

Distributed TLS over Multitask Networks with Adaptive Intertask Cooperation

Network Fault-tolerant and Byzantine-resilient Social Learning via Collaborative Hierarchical Non-Bayesian Learning