Abstract:With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (DRL). In this article, we focus on the task where the agent needs to learn multidimensional deterministic policies to control, which is very common in real scenarios. Recently, the surrogate gradient method has been utilized for training multilayer SNNs, which allows SNNs to achieve comparable performance with the corresponding deep networks in this task. Most existing spike-based reinforcement learning (RL) methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully connected (FC) layer. However, the decimal characteristic of the firing rate brings the floating-point matrix operations to the FC layer, making the whole SNN unable to deploy on the neuromorphic hardware directly. To develop a fully spiking actor network (SAN) without any floating-point matrix operations, we draw inspiration from the nonspiking interneurons found in insects and employ the membrane voltage of the nonspiking neurons to represent the action. Before the nonspiking neurons, multiple population neurons are introduced to decode different dimensions of actions. Since each population is used to decode a dimension of action, we argue that the neurons in each population should be connected in time domain and space domain. Hence, the intralayer connections are used in output populations to enhance the representation capacity. This mechanism exists extensively in animals and has been demonstrated effectively. Finally, we propose a fully SAN with intralayer connections (ILC-SAN). Extensive experimental results demonstrate that the proposed method outperforms the state-of-the-art performance on continuous control tasks from OpenAI gym. Moreover, we estimate the theoretical energy consumption when deploying ILC-SAN on neuromorphic chips to illustrate its high energy efficiency.

A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations

Energy-aware bio-inspired spiking reinforcement learning system architecture for real-time autonomous edge applications

Reinforcement Learning in Spiking Neural Networks with Stochastic and Deterministic Synapses

A Spiking Neural Network Structure Implementing Reinforcement Learning

Deep Reinforcement Learning with Spiking Q-learning

A Low Latency Adaptive Coding Spike Framework for Deep Reinforcement Learning

Human-Level Control Through Directly-Trained Deep Spiking Q-Networks

Toward robust and scalable deep spiking reinforcement learning

Learning fast changing slow in spiking neural networks

A Novel Reinforcement Learning Algorithm Based on Multilayer Memristive Spiking Neural Network With Applications

Fully Spiking Actor Network With Intralayer Connections for Reinforcement Learning

Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning

A Low-Power Actor-Critic Framework Based on Memristive Spiking Neural Network

Training spiking neuronal networks to perform motor control using reinforcement and evolutionary learning

Reinforcement learning using a continuous time actor-critic framework with spiking neurons

TripleBrain: An Edge Neuromorphic Architecture for High-accuracy Single-layer Spiking Neural Network with On-chip Self-organizing and Reinforcement Learning

Reinforcement Learning with Brain-Inspired Modulation can Improve Adaptation to Environmental Changes

Improved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied to ATARI games

Towards biologically plausible model-based reinforcement learning in recurrent spiking networks by dreaming new experiences

Memristor Hardware-Friendly Reinforcement Learning

Spiking network model of the cerebellum as a reinforcement learning machine