On the Hardness of Decentralized Multi-Agent Policy Evaluation under Byzantine Attacks

Hairi,Minghong Fang,Zifan Zhang,Alvaro Velasquez,Jia Liu

2024-09-22

Abstract:In this paper, we study a fully-decentralized multi-agent policy evaluation problem, which is an important sub-problem in cooperative multi-agent reinforcement learning, in the presence of up to $f$ faulty agents. In particular, we focus on the so-called Byzantine faulty model with model poisoning setting. In general, policy evaluation is to evaluate the value function of any given policy. In cooperative multi-agent system, the system-wide rewards are usually modeled as the uniform average of rewards from all agents. We investigate the multi-agent policy evaluation problem in the presence of Byzantine agents, particularly in the setting of heterogeneous local rewards. Ideally, the goal of the agents is to evaluate the accumulated system-wide rewards, which are uniform average of rewards of the normal agents for a given policy. It means that all agents agree upon common values (the consensus part) and furthermore, the consensus values are the value functions (the convergence part). However, we prove that this goal is not achievable. Instead, we consider a relaxed version of the problem, where the goal of the agents is to evaluate accumulated system-wide reward, which is an appropriately weighted average reward of the normal agents. We further prove that there is no correct algorithm that can guarantee that the total number of positive weights exceeds $|\mathcal{N}|-f $, where $|\mathcal{N}|$ is the number of normal agents. Towards the end, we propose a Byzantine-tolerant decentralized temporal difference algorithm that can guarantee asymptotic consensus under scalar function approximation. We then empirically test the effective of the proposed algorithm.

Cryptography and Security,Distributed, Parallel, and Cluster Computing,Machine Learning

What problem does this paper attempt to address?

This paper attempts to solve the difficult problem of policy evaluation in a fully decentralized multi - agent system in the presence of Byzantine attacks. Specifically:

1. **Problem Background**:
- In cooperative multi - agent reinforcement learning (MARL), policy evaluation is an important sub - problem, whose goal is to evaluate the value function of a given policy.
- In an ideal cooperative multi - agent system, all agents need to reach an agreement, and this agreement value should be the uniform average of the cumulative system rewards of normal agents.

2. **Core Challenges**:
- The paper focuses on the situation under the Byzantine faulty model, that is, some agents may send arbitrary or carefully constructed information to disrupt the normal operation of the system.
- Byzantine agents can send inconsistent information to different neighbors during the communication process, which makes the problem more complicated.

3. **Main Research Contents**:
- **Unachievability Proof**: The paper first proves that in the presence of Byzantine agents, it is impossible to accurately evaluate the uniform average of the cumulative system rewards of all normal agents (as shown in the formula below). Therefore, the author considers a relaxed version of the problem, changing the goal to evaluating the weighted average reward.
\[
V(s)=E\left[\sum_{t = 0}^{\infty}\gamma^t\frac{1}{|N|}\sum_{i\in N}r_i(s_t,a_t)\mid s_0 = s\right]
\]
where \(N\) represents the set of normal agents, and \(|N|\) is the number of normal agents.
- **Weight Restriction**: It is further proved that for the relaxed problem, there is no correct algorithm that can guarantee that the number of positive weights exceeds \(|N|-f\), where \(f\) is the maximum number of Byzantine agents.
- **Byzantine - Fault - Tolerant Algorithm**: A Byzantine - tolerant decentralized temporal difference (BDTD) algorithm is proposed, which can ensure asymptotic consistency under scalar function approximation.

4. **Contribution Summary**:
- It is proved that in the presence of Byzantine agents, accurate value function evaluation cannot be achieved.
- It is further proved that for the relaxed weighted average reward problem, the upper limit of the number of positive weights is \(|N|-f\).
- A Byzantine - fault - tolerant decentralized temporal difference algorithm is proposed to ensure that all normal agents can reach an agreement.

Through these studies, the paper reveals the inherent difficulty of policy evaluation in multi - agent systems in the presence of Byzantine attacks and provides a feasible solution.