Abstract:With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (DRL). In this article, we focus on the task where the agent needs to learn multidimensional deterministic policies to control, which is very common in real scenarios. Recently, the surrogate gradient method has been utilized for training multilayer SNNs, which allows SNNs to achieve comparable performance with the corresponding deep networks in this task. Most existing spike-based reinforcement learning (RL) methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully connected (FC) layer. However, the decimal characteristic of the firing rate brings the floating-point matrix operations to the FC layer, making the whole SNN unable to deploy on the neuromorphic hardware directly. To develop a fully spiking actor network (SAN) without any floating-point matrix operations, we draw inspiration from the nonspiking interneurons found in insects and employ the membrane voltage of the nonspiking neurons to represent the action. Before the nonspiking neurons, multiple population neurons are introduced to decode different dimensions of actions. Since each population is used to decode a dimension of action, we argue that the neurons in each population should be connected in time domain and space domain. Hence, the intralayer connections are used in output populations to enhance the representation capacity. This mechanism exists extensively in animals and has been demonstrated effectively. Finally, we propose a fully SAN with intralayer connections (ILC-SAN). Extensive experimental results demonstrate that the proposed method outperforms the state-of-the-art performance on continuous control tasks from OpenAI gym. Moreover, we estimate the theoretical energy consumption when deploying ILC-SAN on neuromorphic chips to illustrate its high energy efficiency.

Mixture of personality improved Spiking actor network for efficient multi-agent cooperation

MAPPO method based on attention behavior network

Decomposing Shared Networks for Separate Cooperation with Multi-Agent Reinforcement Learning

Expert demonstrations guide reward decomposition for multi-agent cooperation

Multi-Sacle Dynamic Coding Improved Spiking Actor Network for Reinforcement Learning

A brain-inspired theory of mind spiking neural network improves multi-agent cooperation and competition

Population-coding and Dynamic-neurons improved Spiking Actor Network for Reinforcement Learning

Adaptive Reward Method for End-to-End Cooperation Based on Multi-agent Reinforcement Learning

Fully Spiking Actor Network With Intralayer Connections for Reinforcement Learning

Biologically-Plausible Topology Improved Spiking Actor Network for Efficient Deep Reinforcement Learning

Multi-Attribute Dynamic Attenuation Learning Improved Spiking Actor Network

ProAgent: Building Proactive Cooperative Agents with Large Language Models

MAR2MIX: A Novel Model for Dynamic Problem in Multi-agent Reinforcement Learning.

Learning Multi-Agent Cooperation via Considering Actions of Teammates

ADMN: Agent-Driven Modular Network for Dynamic Parameter Sharing in Cooperative Multi-Agent Reinforcement Learning

Learning to Cooperate with Humans using Generative Agents

TLMIX: Twin Leader Mixing Network for Cooperative Multi-Agent Reinforcement Learning.

Verco: Learning Coordinated Verbal Communication for Multi-agent Reinforcement Learning

SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents

Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning

Learning to Model Diverse Driving Behaviors in Highly Interactive Autonomous Driving Scenarios with Multi-Agent Reinforcement Learning