DPMAC: Differentially Private Communication for Cooperative Multi-Agent Reinforcement Learning

Canzhe Zhao,Yanjie Ze,Jing Dong,Baoxiang Wang,Shuai Li

2023-08-19

Abstract:Communication lays the foundation for cooperation in human society and in multi-agent reinforcement learning (MARL). Humans also desire to maintain their privacy when communicating with others, yet such privacy concern has not been considered in existing works in MARL. To this end, we propose the \textit{differentially private multi-agent communication} (DPMAC) algorithm, which protects the sensitive information of individual agents by equipping each agent with a local message sender with rigorous $(\epsilon, \delta)$-differential privacy (DP) guarantee. In contrast to directly perturbing the messages with predefined DP noise as commonly done in privacy-preserving scenarios, we adopt a stochastic message sender for each agent respectively and incorporate the DP requirement into the sender, which automatically adjusts the learned message distribution to alleviate the instability caused by DP noise. Further, we prove the existence of a Nash equilibrium in cooperative MARL with privacy-preserving communication, which suggests that this problem is game-theoretically learnable. Extensive experiments demonstrate a clear advantage of DPMAC over baseline methods in privacy-preserving scenarios.

Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the problem of achieving communication in multi-agent reinforcement learning (MARL) while protecting the privacy of individual agents. Specifically, although existing MARL research has demonstrated the importance of communication in promoting agent cooperation, these methods often overlook the potential privacy leakage issues during communication. For example, in autonomous driving systems, each car acts as an agent, and they need to communicate to improve the overall safety and efficiency of the system, but at the same time, they need to protect the personal privacy information of the car owners from being leaked.

To solve this problem, the authors propose an algorithm called "Differential Privacy Multi-Agent Communication" (DPMAC). This algorithm equips each agent with a local message sender and ensures that its communication process meets the requirements of differential privacy (DP). Unlike the method of directly adding noise to the messages, DPMAC employs a random message sender, which mitigates the instability caused by adding DP noise by adjusting the learned message distribution. Additionally, the paper proves the existence of a Nash equilibrium in collaborative MARL with privacy-preserving communication, indicating that such problems are learnable.

Through extensive experimental validation, DPMAC significantly outperforms baseline methods in privacy-preserving scenarios and achieves competitive performance even without privacy constraints. Overall, the main contribution of this paper is the first proposal of a solution for achieving private communication within the MARL framework and demonstrating that this solution is theoretically feasible.