Optimal Synchronization Control for Heterogeneous Multi-Agent Systems: Online Adaptive Learning Solutions

Yuanqiang Zhou,Dewei Li,Furong Gao

DOI: https://doi.org/10.48550/arXiv.2011.05663

2020-11-11

Abstract:This paper presents an online adaptive learning solution to optimal synchronization control problem of heterogeneous multi-agent systems via a novel distributed policy iteration approach.

Systems and Control,Optimization and Control

What problem does this paper attempt to address?

Optimal model-free output synchronization of heterogeneous systems using off-policy reinforcement learning

Hamidreza Modares,Subramanya P. Nageshrao,Gabriel A. Delgado Lopes,Robert Babuška,Frank L. Lewis

DOI: https://doi.org/10.1016/j.automatica.2016.05.017

IF: 6.4

2016-09-01

Automatica

Abstract:This paper considers optimal output synchronization of heterogeneous linear multi-agent systems. Standard approaches to output synchronization of heterogeneous systems require either the solution of the output regulator equations or the incorporation of a p-copy of the leader’s dynamics in the controller of each agent. By contrast, in this paper neither one is needed. Moreover, here both the leader’s and the follower’s dynamics are assumed to be unknown. First, a distributed adaptive observer is designed to estimate the leader’s state for each agent. The output synchronization problem is then formulated as an optimal control problem and a novel model-free off-policy reinforcement learning algorithm is developed to solve the optimal output synchronization problem online in real time. It is shown that this optimal distributed approach implicitly solves the output regulation equations without actually doing so. Simulation results are provided to verify the effectiveness of the proposed approach.

automation & control systems,engineering, electrical & electronic
Distributed Synchronization Control of Nonaffine Multiagent Systems with Guaranteed Performance

Wenchao Meng,Peter Xiaoping Liu,Qinmin Yang,Youxian Sun

DOI: https://doi.org/10.1109/tnnls.2019.2920892

IF: 14.255

2020-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This paper deals with the synchronization control problem in the leader-follower format of a class of high-order nonaffine nonlinear multiagent systems under a directed communication protocol. A novel adaptive neural distributed synchronization scheme with guaranteed performance is proposed. The main contribution lies in the fact that both nonaffine agent dynamics, which basically makes most existing agent dynamics as special cases, and guaranteed synchronization performance are taken into account. The difficulty lies mainly in the nonaffine terms and coupling terms due to the interactions of agents. To overcome this challenge, an augmented quadratic Lyapunov function by incorporating the lower bounds of control gains is proposed. The problems resulting from the nonaffine dynamics and the coupling terms among agents are solved by incorporating the special property of radial basis function neural network into the derivative of the augmented quadratic Lyapunov function. The unknown nonaffine terms are addressed by using an indirected neural network approach. A nonlinear mapping is built to relate the local consensus error to a new one, which is subsequently stabilized via Lyapunov synthesis. As a result, the proposed approach can ensure the outputs of all follower agents to track the outputs of the leader, while the synchronization performance bounds can be quantified on both transient and steady-state stages. All other signals in the closed loop are ensured to be semiglobally, uniformly, and ultimately bounded. Finally, the effectiveness of the proposed controller is verified through a heterogeneous four-agent example.
Optimal self-learning cooperative control for continuous-time heterogeneous multi-agent systems

Wei Qinglai,Liu Derong,Song Ruizhuo

DOI: https://doi.org/10.1109/chicc.2015.7260101

2015-01-01

Abstract:In this paper, an optimal self-learning cooperative control for heterogeneous multi-agent systems by iterative adaptive dynamic programming (ADP) is developed. The main idea is to design an optimal control law by policy iteration based ADP technique which makes all the agents track a given dynamics and simultaneously makes the iterative performance index function reach the Nash equilibrium. The cooperative policy iteration algorithm for graphical differential games is developed to achieve the optimal control law for the agent of each node. Convergence properties are analyzed which make the performance index functions of heterogeneous multi-agent differential graphical games converge to the Nash equilibrium. Simulation example is given to show the effectiveness of the developed optimal self-learning control scheme.
Data-Based Optimal Synchronization Control for Discrete-Time Nonlinear Heterogeneous Multiagent Systems

Hao Fu,Xin Chen,Wei Wang,Min Wu

DOI: https://doi.org/10.1109/tcyb.2020.3004494

IF: 11.8

2022-01-01

IEEE Transactions on Cybernetics

Abstract:This article investigates the optimal synchronization problem for unknown discrete-time nonlinear heterogeneous multiagent systems (MASs). It is very intractable to derive the analytical solutions of coupled Bellman's equations, which are necessary to overcome this problem. We propose a data-based optimal synchronization control strategy based on a hierarchical and distributed optimal control framework composed of a model reference adaptive control (MRAC) layer and a distributed control layer. In the MRAC layer, the similar-offline MRAC algorithm is developed to make subsystems of MASs track their reference models, respectively. Then, the distributed optimal control problem of nonlinear heterogeneous MASs is transformed into that of homogeneous MASs composed of the reference models and the leader. In the distributed control layer, the distributed reference policy iteration algorithm is proposed to derive the solutions of coupled composite nonlinear Bellman's equations, which ensure that the homogeneous MASs reach synchronization with optimum. The suboptimal synchronization control is achieved via optimization further. Convergence analysis of both algorithms is rigorously provided. The simulation results verify the effectiveness of the proposed strategy.
Observer-Based Adaptive Synchronization Control of Unknown Discrete-Time Nonlinear Heterogeneous Systems

Hao Fu,Xin Chen,Wei Wang,Min Wu

DOI: https://doi.org/10.1109/tnnls.2020.3028569

IF: 14.255

2020-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This article is concerned with the optimal synchronization problem for discrete-time nonlinear heterogeneous multiagent systems (MASs) with an active leader. To overcome the difficulty in the derivation of the optimal control protocols for these systems, we develop an observer-based adaptive synchronization control approach, including the designs of a distributed observer and a distributed model reference adaptive controller with no prior knowledge of all agents' dynamics. To begin with, for the purpose of estimating the state of a nonlinear active leader for each follower, an adaptive neural network distributed observer is designed. Such an observer serves as a reference model in the distributed model reference adaptive control (MRAC). Then, a reinforcement learning-based distributed MRAC algorithm is presented to make every follower track its corresponding reference model on behavior in real time. In this algorithm, a distributed actor-critic network is employed to approximate the optimal distributed control protocols and the cost function. Through convergence analysis, the overall observer estimation error, the model reference tracking error, and the weight estimation errors are proved to be uniformly ultimately bounded. The developed approach further achieves the synchronization by means of synthesizing these results. The effectiveness of the developed approach is verified through a numerical example.
Optimal Antisynchronization Control for Unknown Multiagent Systems with Deep Deterministic Policy Gradient Approach.

Cuijuan Zhang,Lianghao Ji,Shasha Yang,Huaqing Li

DOI: https://doi.org/10.1016/j.ins.2022.12.008

IF: 8.1

2022-01-01

Information Sciences

Abstract:The topic of optimal antisynchronization control for unknown multiagent systems (MASs) with cooperative-competitive (i.e.,coopetition) interaction is challenging because of the complex connection characteristics and the coupling among agents. This paper proposes a reinforcement learning Algorithm based on coopetition strength (CS) to achieve optimal antisynchronization in unknown MASs. Firstly, an innovative CS function is explored, and the local state error information of the agents can be redefined. Furthermore, we propose a novel strategy iteration method to approximate the agent’s optimal control policy. Meanwhile, the proposed algorithm’s convergence analysis is presented, which is based on the Lyapunov stability and functional theorem. In the implementation of a data-based control policy, a network structure of actor-critic (AC) is designed. To improve the robustness of the control policy, the target network and experience replay (ER) are introduced in the training process. Finally, the algorithm’s effectiveness is validated using comparable numerical simulations.
Pareto-optimal Synchronization Control of Nonlinear Multi-Agent Systems Via Integral Reinforcement Learning

Yaning Guo,Qi Sun,Quan Pan,Yintao Wang

DOI: https://doi.org/10.1007/s11071-024-10461-w

IF: 5.741

2024-01-01

Nonlinear Dynamics

Abstract:This paper investigates a class of cooperative synchronization control problem of networked multi-agent systems (MASs) with unknown nonlinear dynamics. It is shown that the synchronization control problem of MASs can be formulated as the cooperative differential graphical game, in which the objective of each agent is to minimize the total team cost in a cooperative manner. The theoretical derivation and established conditions are then provided to achieve Pareto optimality for the cooperative differential graphical games. Furthermore, an adaptive recursive multi-agent reinforcement learning algorithm is proposed to learn a generalized optimal team value function and the associated decentralized Pareto-optimal strategies, where a Nash bargaining strategy and a recursive Bayesian inference scheme with different forgetting factors are respectively developed. Finally, two simulation examples are presented to verify effectiveness of the proposed learning-based synchronization control approach.
An Iterative Learning Control Approach For Synchronization Of Multi-Agent Systems Under Iteration-Varying Graph

Shiping Yang,Jian-Xin Xu,Miao Yu

DOI: https://doi.org/10.1109/CDC.2013.6760947

2013-01-01

Abstract:In this work, an iterative learning control (ILC) strategy is applied to synchronize the outputs from a group of homogeneous agents under iteration-varying communication topology. First, we show that the ILC strategy works for fixed strongly connected graph, which lays out the analysis framework for the rest developments. Next, the result is extended to iteration-varying topology, where the graph is strongly connected in each iteration. Then, the result is further generalized to uniformly strongly connected graph along the iteration domain. Matrix norm properties together with contraction mapping based analysis are utilized to prove the results. Finally, a numerical example is presented to verify the obtained results.
Data-Based Optimal Synchronization of Heterogeneous Multiagent Systems in Graphical Games Via Reinforcement Learning.

Chunping Xiong,Qian Ma,Jian Guo,Frank L. Lewis

DOI: https://doi.org/10.1109/tnnls.2023.3291542

IF: 14.255

2023-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This article studies the optimal synchronization of linear heterogeneous multiagent systems (MASs) with partial unknown knowledge of the system dynamics. The object is to realize system synchronization as well as minimize the performance index of each agent. A framework of heterogeneous multiagent graphical games is formulated first. In the graphical games, it is proved that the optimal control policy relying on the solution of the Hamilton-Jacobian-Bellmen (HJB) equation is not only in Nash equilibrium, but also the best response to fixed control policies of its neighbors. To solve the optimal control policy and the minimum value of the performance index, a model-based policy iteration (PI) algorithm is proposed. Then, according to the model-based algorithm, a data-based off-policy integral reinforcement learning (IRL) algorithm is put forward to handle the partially unknown system dynamics. Furthermore, a single-critic neural network (NN) structure is used to implement the data-based algorithm. Based on the data collected by the behavior policy of the data-based off-policy algorithm, the gradient descent method is used to train NNs to approach the ideal weights. In addition, it is proved that all the proposed algorithms are convergent, and the weight-tuning law of the single-critic NNs can promote optimal synchronization. Finally, a numerical example is proposed to show the effectiveness of the theoretical analysis.
Synchronizability-Based Distributed Learning Control for Multi-Agent Systems

Yuxin Wu,Deyuan Meng

DOI: https://doi.org/10.1109/tcsii.2023.3332352

2024-01-01

Abstract:In this brief, the distributed learning control problem is addressed for multi-agent systems with irregular dynamics by focusing on the transient synchronization objective. First, the synchronizability property is defined to reveal whether the multi-agent system has the ability to reach the transient synchronization objective, for which a synchronizability criterion is developed in a distributed manner. Then, given the synchronizability property, a distributed learning control protocol is further proposed through the full usage of the experience information from neighbor agents. Under the quasi-strong connectivity, the necessary and sufficient condition is exploited to accomplish the transient synchronization objective for the multi-agent system despite whether it is regular. In particular, the specific selection condition on the gain matrix is presented to provide a fully distributed design manner for the proposed distributed learning control protocol.
Data-Driven Optimal Synchronization Control for Leader-Follower Multiagent Systems

Yuanqiang Zhou,Dewei Li,Furong Gao

DOI: https://doi.org/10.1109/tsmc.2022.3184001

2023-01-01

Abstract:In this article, we develop data-driven optimal synchronization control architectures for leader-follower multiagent systems with additive disturbances and unknown system matrices. To minimize output synchronization error, algebraic Riccati equations (AREs) are derived, and unique feedback gains are determined by policy iteration. On that basis, two data-driven optimal synchronization control algorithms are developed without relying on the dynamics of the system, which guarantee output synchronization while minimizing synchronization errors and rejecting disturbances. The first algorithm uses the output synchronization error data to perform online data-driven learning (DDL), while the second algorithm uses the input data to perform DDL, where both data sample requirements are transformed into rank conditions. We have presented rigorous theoretical analyses of our proposed algorithms, which demonstrate that if an initial control protocol can make the system achieve output synchronization under mild conditions, our proposed two algorithms can take advantage of the data from reaching synchronization to optimize the closed-loop performance. Finally, a numerical example is provided to emphasize the effectiveness of our methods.
Transient Bipartite Synchronization for Cooperative-Antagonistic Multiagent Systems with Switching Topologies

Yuxin Wu,Deyuan Meng,Zheng-Guang Wu

DOI: https://doi.org/10.1109/tcyb.2021.3070402

IF: 11.8

2022-01-01

IEEE Transactions on Cybernetics

Abstract:This article aims at addressing the transient bipartite synchronization problem for cooperative-antagonistic multiagent systems with switching topologies. A distributed iterative learning control protocol is presented for agents by resorting to the local information from their neighbor agents. Through learning from other agents, the control input of each agent is updated iteratively such that the transient bipartite synchronization can be achieved over the targeted finite horizon under the simultaneously structurally balanced signed digraph. To be specific, all agents finally have the same output moduli at each time instant over the desired finite-time interval, which overcomes the influences caused by the antagonisms among agents and topology nonrepetitiveness along the iteration axis. As a counterpart, it is revealed that the stability can be achieved over the targeted finite horizon in the presence of a constantly structurally unbalanced signed digraph. Simulation examples are carried out to demonstrate the effectiveness of the distributed learning results developed among multiple agents.
Optimal Model-Free Output Synchronization of Heterogeneous Multiagent Systems under Switching Topologies

Chaoxu Mu,Qian Zhao,Changyin Sun

DOI: https://doi.org/10.1109/tie.2019.2958277

IF: 7.7

2020-01-01

IEEE Transactions on Industrial Electronics

Abstract:In this article, the output synchronization of discrete-time heterogeneous multiagent systems over directed switching topologies is investigated. For output synchronization problem, the optimal control protocols for all agents depend on the solutions of a series of algebraic Riccati equations (AREs), which are difficult to be solved analytically. Besides, both the agents' and the leader's dynamics are supposed to be unknown. The distributed adaptive observer for each agent is designed first to estimate the leader's state without requiring complete knowledge of the leader's dynamics. Based on the trained observers, the output synchronization problem is formulated to an optimal control problem. An observer-based Q-learning algorithm is developed to solve the AREs using system data rather than the accurate system models. The optimal analytic distributed control policies can be obtained by policy iteration combined least square method. It is proved theoretically that the output synchronization is ensured based on the distributed adaptive observers under switching topologies. Ultimately, the theoretical results are demonstrated via three simulation examples.
Adaptive Optimal Bipartite Consensus Control for Heterogeneous Multi-Agent Systems

Bingyun Liang,Yanling Wei,Wenwu Yu

DOI: https://doi.org/10.1109/TCNS.2024.3395724

IF: 4.347

2024-01-01

IEEE Transactions on Control of Network Systems

Abstract:In this article, a distributed adaptive optimal control method is proposed to solve the bipartite consensus problem for heterogeneous multi-agent systems. First, with a change in coordinates, the optimal bipartite consensus can be transformed into the optimal consensus problem and the optimal control law is also established. Second, a distributed state observer is designed for each agent to estimate the leader's state under the cooperate/antagonistic interaction, which is used to replace the unavailable leader's signal. Then, to find the optimal control solution adaptively, two online integral reinforcement learning algorithms, i.e., on-policy and off-policy, are developed. Based on the policy iteration in learning process, the algorithms proposed here utilize the state data of systems without requiring the complete knowledge of the leader's and agents' dynamics. It is proven that the observer is exponentially convergent, which guarantees the accuracy of solution in algorithms. Finally, two examples are given to show the validity of the proposed method.
Distributed optimal output synchronization of heterogeneous multiagent systems with communication delays

Wei Zhao,Huaipin Zhang,Wenwu Yu

DOI: https://doi.org/10.1002/rnc.7369

IF: 3.8973

2024-04-18

International Journal of Robust and Nonlinear Control

Abstract:The optimal output synchronization problem of heterogeneous multi‐agent systems (HMASs) with communication delays is studied in this article. In order to compensate for communication delays actively, we employ networked data‐driven predictive control method to predict the agents' outputs. Then the output synchronization issue of HAMSs is transformed into the stabilization issue of the local pseudo systems, and model‐free off‐policy reinforcement learning technique is used to learn optimal control policies of the local systems. Combined predicted output information with optimal control policies, we develop a distributed data‐driven optimal output policy for each follower. Finally, a numerical example is given to show the effectiveness of our results.

automation & control systems,engineering, electrical & electronic,mathematics, applied
Decentralized Adaptive Synchronization Control For Networked Multi-Agent Systems With Unknown Parameter

Lianzeng Ma,Huaguang Zhang,Xuebo Chen

DOI: https://doi.org/10.1109/CAC.2013.6775774

2013-01-01

Abstract:This paper addresses adaptive synchronization control between two Networked Multi-agent Systems with nonidentical topological structures and unknown parameters. A novel controller with adaptive coordinator was designed in order to realize synchronization control. Applying decentralized control policy, coordinator with adaptive parameter estimation was designed to obtain stable desired state by adjusting interconnection level. Error control system asymptotic stability was proved. The designed controller can easily be implemented in practice. An illustrative example is given to demonstrate the effectiveness of the present method.
Optimal Synchronization Control of Multiagent Systems with Input Saturation Via Off-Policy Reinforcement Learning.

Jiahu Qin,Man Li,Yang Shi,Qichao Ma,Wei Xing Zheng

DOI: https://doi.org/10.1109/tnnls.2018.2832025

IF: 14.255

2019-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:In this paper, we aim to investigate the optimal synchronization problem for a group of generic linear systems with input saturation. To seek the optimal controller, Hamilton-Jacobi-Bellman (HJB) equations involving nonquadratic input energy terms in coupled forms are established. The solutions to these coupled HJB equations are further proven to be optimal and the induced controllers constitute interactive Nash equilibrium. Due to the difficulty to analytically solve HJB equations, especially in coupled forms, and the possible lack of model information of the systems, we apply the data-based off-policy reinforcement learning algorithm to learn the optimal control policies. A byproduct of this off-policy algorithm is shown that it is insensitive to probing noise that is exerted to the system to maintain persistence of excitation condition. In order to implement this off-policy algorithm, we employ actor and critic neural networks to approximate the controllers and the cost functions. Furthermore, the estimated control policies obtained by this presented implementation are proven to converge to the optimal ones under certain conditions. Finally, an illustrative example is provided to verify the effectiveness of the proposed algorithm.
Optimal Synchronization Control of Heterogeneous Asymmetric Input-Constrained Unknown Nonlinear MASs via Reinforcement Learning

Lina Xia,Qing Li,Ruizhuo Song,Hamidreza Modares

DOI: https://doi.org/10.1109/JAS.2021.1004359

2022-01-01

IEEE/CAA Journal of Automatica Sinica

Abstract:The asymmetric input-constrained optimal synchronization problem of heterogeneous unknown nonlinear multiagent systems (MASs) is considered in the paper. Intuitively, a state-space transformation is performed such that satisfaction of symmetric input constraints for the transformed system guarantees satisfaction of asymmetric input constraints for the original system. Then, considering that the leader's information is not available to every follower, a novel distributed observer is designed to estimate the leader's state using only exchange of information among neighboring followers. After that, a network of augmented systems is constructed by combining observers and followers dynamics. A nonquadratic cost function is then leveraged for each augmented system (agent) for which its optimization satisfies input constraints and its corresponding constrained Hamilton-Jacobi-Bellman (HJB) equation is solved in a data-based fashion. More specifically, a data-based off-policy reinforcement learning (RL) algorithm is presented to learn the solution to the constrained HJB equation without requiring the complete knowledge of the agents' dynamics. Convergence of the improved RL algorithm to the solution to the constrained HJB equation is also demonstrated. Finally, the correctness and validity of the theoretical results are demonstrated by a simulation example.
Optimal output synchronization of heterogeneous multi-agent systems using measured input-output data

Yingying Liu,Zhanshan Wang

DOI: https://doi.org/10.1016/j.ins.2021.08.044

IF: 8.1

2022-01-01

Information Sciences

Abstract:In this paper, the optimal output synchronization of heterogeneous multi-agent systems is studied. To overcome the shortcoming of previous methods that require system model, a data-based optimal control policy is proposed for the output synchronization problem. Besides, the intrinsic equivalence relationship between model-based optimal solution and the proposed data-based optimal solution is proved. To solve the proposed control policy, a reinforcement learning algorithm based on measured input-output data is presented, which is different from existing model-free algorithms based on internal state. According to the algorithm, the optimal output synchronization problem of heterogeneous multi-agent systems can be solved when the full-state vector is unavailable. Finally, the effectiveness of proposed algorithm is verified by simulation examples.
Heuristic Dynamic Programming-Based Learning Control for Discrete-Time Disturbed Multi-Agent Systems

Zhang Yao,Mu Chaoxu,Zhang Yong,Feng Yanghe

DOI: https://doi.org/10.1007/s11768-021-00049-9

2021-01-01

Control Theory and Technology

Abstract:Owing to extensive applications in many fields, the synchronization problem has been widely investigated in multi-agent systems. The synchronization for multi-agent systems is a pivotal issue, which means that under the designed control policy, the output of systems or the state of each agent can be consistent with the leader. The purpose of this paper is to investigate a heuristic dynamic programming (HDP)-based learning tracking control for discrete-time multi-agent systems to achieve synchronization while considering disturbances in systems. Besides, due to the difficulty of solving the coupled Hamilton–Jacobi–Bellman equation analytically, an improved HDP learning control algorithm is proposed to realize the synchronization between the leader and all following agents, which is executed by an action-critic neural network. The action and critic neural network are utilized to learn the optimal control policy and cost function, respectively, by means of introducing an auxiliary action network. Finally, two numerical examples and a practical application of mobile robots are presented to demonstrate the control performance of the HDP-based learning control algorithm.

Optimal Synchronization Control for Heterogeneous Multi-Agent Systems: Online Adaptive Learning Solutions

Optimal model-free output synchronization of heterogeneous systems using off-policy reinforcement learning

Distributed Synchronization Control of Nonaffine Multiagent Systems with Guaranteed Performance

Optimal self-learning cooperative control for continuous-time heterogeneous multi-agent systems

Data-Based Optimal Synchronization Control for Discrete-Time Nonlinear Heterogeneous Multiagent Systems

Observer-Based Adaptive Synchronization Control of Unknown Discrete-Time Nonlinear Heterogeneous Systems

Optimal Antisynchronization Control for Unknown Multiagent Systems with Deep Deterministic Policy Gradient Approach.

Pareto-optimal Synchronization Control of Nonlinear Multi-Agent Systems Via Integral Reinforcement Learning

An Iterative Learning Control Approach For Synchronization Of Multi-Agent Systems Under Iteration-Varying Graph

Data-Based Optimal Synchronization of Heterogeneous Multiagent Systems in Graphical Games Via Reinforcement Learning.

Synchronizability-Based Distributed Learning Control for Multi-Agent Systems

Data-Driven Optimal Synchronization Control for Leader-Follower Multiagent Systems

Transient Bipartite Synchronization for Cooperative-Antagonistic Multiagent Systems with Switching Topologies

Optimal Model-Free Output Synchronization of Heterogeneous Multiagent Systems under Switching Topologies

Adaptive Optimal Bipartite Consensus Control for Heterogeneous Multi-Agent Systems

Distributed optimal output synchronization of heterogeneous multiagent systems with communication delays

Decentralized Adaptive Synchronization Control For Networked Multi-Agent Systems With Unknown Parameter

Optimal Synchronization Control of Multiagent Systems with Input Saturation Via Off-Policy Reinforcement Learning.

Optimal Synchronization Control of Heterogeneous Asymmetric Input-Constrained Unknown Nonlinear MASs via Reinforcement Learning

Optimal output synchronization of heterogeneous multi-agent systems using measured input-output data

Heuristic Dynamic Programming-Based Learning Control for Discrete-Time Disturbed Multi-Agent Systems