Abstract:Deep reinforcement learning (DRL) combines reinforcement learning algorithms with deep neural networks (DNNs). Spiking neural networks (SNNs) have been shown to be a biologically plausible and energy efficient alternative to DNNs. Since the introduction of surrogate gradient approaches that allowed to overcome the discontinuity in the spike function, SNNs can now be trained with the backpropagation through time (BPTT) algorithm. While largely explored on supervised learning problems, little work has been done on investigating the use of SNNs as function approximators in DRL. Here we show how SNNs can be applied to different DRL algorithms like Deep Q-Network (DQN) and Twin-Delayed Deep Deteministic Policy Gradient (TD3) for discrete and continuous action space environments, respectively. We found that SNNs are sensitive to the additional hyperparameters introduced by spiking neuron models like current and voltage decay factors, firing thresholds, and that extensive hyperparameter tuning is inevitable. However, we show that increasing the simulation time of SNNs, as well as applying a two-neuron encoding to the input observations helps reduce the sensitivity to the membrane parameters. Furthermore, we show that randomizing the membrane parameters, instead of selecting uniform values for all neurons, has stabilizing effects on the training. We conclude that SNNs can be utilized for learning complex continuous control problems with state-of-the-art DRL algorithms. While the training complexity increases, the resulting SNNs can be directly executed on neuromorphic processors and potentially benefit from their high energy efficiency.

The State of Sparse Training in Deep Reinforcement Learning

Dynamic Sparse Training for Deep Reinforcement Learning

S2RL: DoWe Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch

Learning Sparse Representations Incrementally in Deep Reinforcement Learning

Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorations

The Utility of Sparse Representations for Control in Reinforcement Learning

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

Deep Model-Based Reinforcement Learning for Predictive Control of Robotic Systems with Dense and Sparse Rewards

Dealing with Sparse Rewards in Reinforcement Learning

Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization

Dynamic sparse coding-based value estimation network for deep reinforcement learning

A Novel Topology Adaptation Strategy for Dynamic Sparse Training in Deep Reinforcement Learning.

A Data-efficiency Training Framework for Deep Reinforcement Learning

State Representation Learning for Effective Deep Reinforcement Learning.

Toward robust and scalable deep spiking reinforcement learning

Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning

A Data-Efficient Training Method for Deep Reinforcement Learning

Dealing with Sparse Rewards Using Graph Neural Networks

DQN with model-based exploration: efficient learning on environments with sparse rewards